Add multi-page PDF support, expand USA tier to 219 species, add all 4 vignettes

- Fix app_database_prep to ingest all pages of multi-page PDFs (not just first page)
- Expand USA IPM tier from 110 to 219 species
- Add all 4 example vignettes: SLF, Striga, FAW, Sahara Mustard
- Improve RAG prompt to be more concise and avoid document number references
- Update README with chunk counts, new species pipeline docs, and updated eval metrics
- Rebuild vector database and regenerate evaluation results

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

README.md

@@ -65,9 +65,8 @@ OPENROUTER_API_KEY=...        # optional, for Llama/Gemini
 ├── examples-work-out/
 │   ├── generate_latex.py       # JSON → LaTeX converter for paper examples
 │   ├── example_results_*.json  # Generated example outputs
-│   ├── example_results_20260301_191932.json  # Striga asiatica (Africa + USA) — used in paper supplementary
-│   ├── example_results_20260302_102648.json  # Spodoptera frugiperda (India + USA) — used in paper supplementary
-│   └── species-verification.md # Example vignettes motivation (flow is approximate, not exact)
+│   ├── example_results_20260303_110427.json  # All 4 vignettes (SLF, Striga, FAW, Sahara Mustard) — latest, updated prompt
+│   └── species-verification.md # Vignette tracking: species, regions, DB status, example outputs
 │
 └── writing/
     └── 65d4fadc59fceb1a54d1aae6/main.tex  # Paper draft
@@ -75,20 +74,21 @@ OPENROUTER_API_KEY=...        # optional, for Llama/Gemini
 
 ### Database Build Flow (4 Geographic Tiers)
 
-| Tier | Species | Source |
-|------|---------|--------|
-| Midwest USA | 80 | ISU Handbook PDFs |
-| USA | 110 | GPT-4o generated IPM |
-| Africa | 35 | Expert-curated Excel |
-| India | 11 | Expert-curated Excel |
+| Tier | Species | Chunks | Source |
+|------|---------|--------|--------|
+| Midwest USA | 82 | 388 | ISU Handbook PDFs |
+| USA | 219 | 913 | GPT-4o generated IPM |
+| Africa | 35 | 70 | Expert-curated Excel |
+| India | 11 | 39 | Expert-curated Excel |
+| **Total** | **128 unique** | **1410** | |
 
-**Midwest USA Data (80 species):**
+**Midwest USA Data (82 species):**
 1. PDFs loaded from `agllm-data/agllm-data-isu-field-{insects,weeds}-all-species/` (content source)
 2. `matched_species_results_v2.csv` in each folder maps PDF filename → species name (metadata); region hardcoded as `"Midwest USA"` in `app_database_prep.py`
 3. The `Midwest USA` sheet in the Excel is a reference copy only — **not** read by `app_database_prep.py`
 
-**USA Data (110 species - LLM generated):**
-4. Run `generate_usa_ipm_info.py` to query GPT-4o for all species
+**USA Data (219 species - LLM generated):**
+4. Run `generate_usa_ipm_info.py` to query GPT-4o for all species (species from all 3 primary tiers, filtered to those confirmed in USA)
 5. Creates `USA` sheet in `species-organized/PestID Species - Organized.xlsx` — both content (IPM Info column) and metadata are read from this sheet
 
 **Africa/India Data (35 + 11 species):**
@@ -111,15 +111,57 @@ python generate_usa_ipm_info.py --step process   # Call GPT-4o API
 python generate_usa_ipm_info.py --step parse     # Create Excel sheet
 ```
 
-**Output:** Updates `species-organized/PestID Species - Organized.xlsx` with "USA" sheet containing 110 species present in the United States (pests + beneficials).
+**Output:** Updates `species-organized/PestID Species - Organized.xlsx` with "USA" sheet containing 219 species present in the United States (pests + beneficials).
 
 ### Evaluation Filters (retrieval_evaluation.py)
 | Filter | P@1 | P@2 | P@4 | P@8 | nDCG@1 | nDCG@2 | nDCG@4 | nDCG@8 |
 |--------|-----|-----|-----|-----|--------|--------|--------|--------|
-| No Filter | 0.64 | 0.76 | 0.81 | 0.85 | 0.64 | 0.72 | 0.74 | 0.75 |
-| Species Only | 0.68 | 0.84 | 0.93 | **1.00** | 0.68 | 0.78 | 0.83 | 0.85 |
-| Region Only | 0.69 | 0.78 | 0.84 | 0.87 | 0.69 | 0.75 | 0.78 | 0.79 |
-| Species + Region | **0.79** | **0.91** | **0.99** | **1.00** | **0.79** | **0.87** | **0.90** | **0.91** |
+| No Filter | 0.50 | 0.72 | 0.79 | 0.87 | 0.50 | 0.64 | 0.67 | 0.70 |
+| Species Only | 0.69 | 0.85 | 0.97 | **1.00** | 0.69 | 0.79 | 0.85 | 0.86 |
+| Region Only | 0.53 | 0.77 | 0.84 | 0.89 | 0.53 | 0.68 | 0.72 | 0.73 |
+| Species + Region | **0.74** | **0.89** | **0.99** | **1.00** | **0.74** | **0.83** | **0.88** | **0.89** |
+
+### Adding a New Species (Full Pipeline)
+
+```bash
+# Activate environment (always deactivate first to avoid path conflicts)
+source ~/miniconda3/etc/profile.d/conda.sh && conda deactivate && conda activate agllm-june-15
+
+# Set API keys (or ensure .env file is correct — note: shell env vars override .env)
+export OPENAI_API_KEY="sk-proj-..."
+export OPENROUTER_API_KEY="sk-or-v1-..."
+
+# 1. Copy PDF to the appropriate data directory
+cp "Species Name.pdf" agllm-data/agllm-data-isu-field-{insects,weeds}-all-species/raw-pdfs/
+
+# 2. Add entry to matched_species_results_v2.csv (in the same data directory)
+echo "Genus species,Common Name,Genus species.pdf" >> agllm-data/agllm-data-isu-field-{insects,weeds}-all-species/matched_species_results_v2.csv
+
+# 3. (Optional) Add to Midwest USA sheet in Excel for reference
+#    Edit species-organized/PestID Species - Organized.xlsx
+
+# 4. Regenerate USA tier IPM info (includes new species)
+python generate_usa_ipm_info.py --force
+
+# 5. Rebuild ChromaDB
+python app_database_prep.py
+
+# 6. Run examples to verify
+python app.py --examples
+
+# 7. Regenerate visualization and copy to paper figures
+python species-organized/species_analysis.py
+cp species-organized/species_analysis.pdf writing/65d4fadc59fceb1a54d1aae6/figures-in-making/species_analysis.pdf
+
+# 8. Run retrieval evaluation
+python retrieval_evaluation.py
+```
+
+**Important notes:**
+- Always `conda deactivate && conda activate agllm-june-15` before running (avoids env path conflicts where `agllm-env1-updates-1` python gets used instead)
+- Shell env vars override `.env` file — if you see `Incorrect API key` errors, check `echo $OPENAI_API_KEY`
+- The `matched_species_results_v2.csv` maps PDF filenames → species names; this is the authoritative source for Midwest USA tier
+- Multi-page PDFs are fully supported (all pages ingested and chunked)
 
 ---
 

app.py

@@ -418,10 +418,11 @@ Context retrieved from the database (organized by region):
 {{context}}
 
 Instructions:
-- Answer based ONLY on the provided context. If no context exists for a region, do not invent information for it.
+- Answer based ONLY on the provided context. Do NOT add information beyond what is in the context.
+- Be concise: use bullet points and keep the response brief. Do not repeat similar information across regions.
 - Start with information from the selected region ('{region}') if available. If not available, clearly state that.
-- Then include relevant information from other regions, clearly labeling which region each piece of information comes from.
-- If the context is insufficient to fully answer the question, you may add brief general knowledge, clearly marked as such.
+- Then include relevant information from other regions, labeling by region name (e.g., "According to Midwest USA guidance..."). Do NOT use "Document 1/2/3" references.
+- If the context is insufficient, state what is available and what is missing.
 - {language_constraint}
 
 Question: {{question}}
@@ -672,10 +673,16 @@ def run_examples():
     from datetime import datetime
 
     examples = [
-        # {"species": "Striga asiatica", "region": "Africa", "model": "Llama-3 70B", "mode": "Researcher", "question": "What is the most effective way to manage Striga in my maize field?"},
-        # {"species": "Striga asiatica", "region": "USA",    "model": "Llama-3 70B", "mode": "Researcher", "question": "What is the most effective way to manage Striga in my maize field?"},
+        # Vignette 1: Spotted Lanternfly — Midwest USA (ISU PDF)
+        {"species": "Lycorma delicatula", "region": "Midwest USA", "model": "Llama-3 70B", "mode": "Researcher", "question": "How can I identify and report spotted lanternfly in Iowa?"},
+        # Vignette 2: Witchweed — Africa vs USA cross-regional
+        {"species": "Striga asiatica", "region": "Africa", "model": "Llama-3 70B", "mode": "Researcher", "question": "What is the most effective way to manage Striga in my maize field?"},
+        {"species": "Striga asiatica", "region": "USA",    "model": "Llama-3 70B", "mode": "Researcher", "question": "What is the most effective way to manage Striga in my maize field?"},
+        # Vignette 3: Fall Armyworm — India vs USA cross-regional
         {"species": "Spodoptera frugiperda", "region": "India", "model": "Llama-3 70B", "mode": "Researcher", "question": "What are the recommended IPM strategies for managing fall armyworm in my corn field?"},
         {"species": "Spodoptera frugiperda", "region": "USA",   "model": "Llama-3 70B", "mode": "Researcher", "question": "What are the recommended IPM strategies for managing fall armyworm in my corn field?"},
+        # Vignette 4: Sahara Mustard — Midwest USA (extension PDF)
+        {"species": "Brassica tournefortii", "region": "Midwest USA", "model": "Llama-3 70B", "mode": "Researcher", "question": "What mechanical, chemical, and cultural control methods are recommended for managing Sahara mustard infestations?"},
     ]
 
     results = []

app_database_prep.py

@@ -128,18 +128,19 @@ excel_file_path = "species-organized/PestID Species - Organized.xlsx"
 ## Process PDF documents using CSV → PDF approach
 print("--- Processing PDF Documents (CSV → PDF approach) ---")
 
-# Function to find PDF file for a given filename
-def find_pdf_file(filename, documents):
-    """Find a PDF document by filename in the loaded documents"""
+# Function to find ALL pages of a PDF file for a given filename
+def find_pdf_pages(filename, documents):
+    """Find all document pages matching a PDF filename in the loaded documents"""
+    matching_docs = []
     for doc in documents:
         doc_filename = doc.metadata["source"].split('/')[-1]
         # Try exact match first
         if doc_filename.lower() == filename.lower():
-            return doc
+            matching_docs.append(doc)
         # Try without extension
-        if doc_filename.lower().replace('.pdf', '') == filename.lower().replace('.pdf', ''):
-            return doc
-    return None
+        elif doc_filename.lower().replace('.pdf', '') == filename.lower().replace('.pdf', ''):
+            matching_docs.append(doc)
+    return matching_docs
 
 pdf_documents_for_splitting = []
 processed_files = set()
@@ -151,27 +152,30 @@ for index, row in metadata_raw.iterrows():
     filename = row['File Name']
     species = row['Species']
 
-    # Find the corresponding PDF document
-    pdf_doc = find_pdf_file(filename, documents)
-
-    if pdf_doc is not None:
-        # Only process if we haven't already processed this file
-        doc_source = pdf_doc.metadata["source"]
-        if doc_source not in processed_files:
-            # Add region for PDF docs (Midwest USA - from ISU Handbook)
-            pdf_doc.metadata["region"] = "Midwest USA"
-
-            # Add species metadata - guaranteed to exist since we're starting from CSV
-            pdf_doc.metadata["matched_specie_0"] = species
+    # Find ALL pages of the corresponding PDF document
+    pdf_pages = find_pdf_pages(filename, documents)
 
+    if pdf_pages:
+        # Use the base filename (without page info) to track processed files
+        base_source = pdf_pages[0].metadata["source"].split('/')[-1].lower()
+        if base_source not in processed_files:
             # Check if there are multiple species for the same file
             same_file_species = metadata_raw[metadata_raw["File Name"].str.lower() == filename.lower()]["Species"]
-            for specie_index, specie_name in enumerate(same_file_species):
-                pdf_doc.metadata[f"matched_specie_{specie_index}"] = specie_name
 
-            pdf_documents_for_splitting.append(pdf_doc)
-            processed_files.add(doc_source)
-            print(f"✓ Processed: {filename} → {species}")
+            for pdf_doc in pdf_pages:
+                # Add region for PDF docs (Midwest USA - from ISU Handbook)
+                pdf_doc.metadata["region"] = "Midwest USA"
+
+                # Add species metadata - guaranteed to exist since we're starting from CSV
+                pdf_doc.metadata["matched_specie_0"] = species
+
+                for specie_index, specie_name in enumerate(same_file_species):
+                    pdf_doc.metadata[f"matched_specie_{specie_index}"] = specie_name
+
+                pdf_documents_for_splitting.append(pdf_doc)
+
+            processed_files.add(base_source)
+            print(f"✓ Processed: {filename} → {species} ({len(pdf_pages)} pages)")
         else:
             print(f"⚠ Already processed: {filename}")
     else:

examples-work-out/example_results_20260302_202103.json

@@ -0,0 +1,20 @@
+[
+  {
+    "species": "Lycorma delicatula",
+    "region": "Midwest USA",
+    "model": "Llama-3 70B",
+    "mode": "Researcher",
+    "question": "How can I identify and report spotted lanternfly in Iowa?",
+    "availability": "**Common Name:** Unknown\n\nInformation for **Lycorma delicatula** is available in region(s): **Midwest USA**.",
+    "answer": "Based on the provided context from the Midwest USA region, here's how you can identify and report spotted lanternfly in Iowa:\n\n**Identification:**\n\n* Egg stage: Look for egg masses, which would be present in Iowa from late July to the following spring.\n* Nymph stage: Identify by their striking colors. Young nymphs are black with white spots, while older nymphs are red and black with white spots. They are often found on host trees, usually on branches, and will move quickly to avoid being seen.\n* Adult stage: Recognize adults by their pinkish-gray color with black spots, long grayish-black legs, and a second pair of red, white, and black wings beneath the gray spotted wings.\n\n**Reporting:**\n\nWhile there is no specific information on reporting in the provided context, as a general knowledge, it is essential to report any spotted lanternfly sightings to the Iowa Department of Agriculture and Land Stewardship or local authorities to help track the spread of this invasive species and inform management efforts.\n\n**Additional information:**\n\nFrom the same region (Midwest USA), it is known that spotted lanternfly feed on a wide variety of plants, but prefer tree of heaven (Ailanthus altissima). This information can be useful when monitoring for the species.\n\nNo additional information on identification and reporting is available from other regions (USA, Africa, or India) in the provided context."
+  },
+  {
+    "species": "Brassica tournefortii",
+    "region": "Midwest USA",
+    "model": "Llama-3 70B",
+    "mode": "Researcher",
+    "question": "What are the most effective methods to control Sahara mustard before it spreads?",
+    "availability": "**Common Name:** Unknown\n\nInformation for **Brassica tournefortii** is available in region(s): **Midwest USA**.",
+    "answer": "Based on the provided context from the Midwest USA region, there is no specific information on the most effective methods to control Sahara mustard. However, it is mentioned that \"it is imperative that further establishment of this invasive weed be prevented and that existing plants be eliminated\" (Document 2 from Midwest USA), indicating the importance of control measures.\n\nUnfortunately, there is no information on control methods in the provided context from the Midwest USA region. \n\nIf we look at other regions, there is no relevant information on control methods in the provided context from the USA, Africa, or India regions.\n\nAs a general knowledge addition, it is known that effective control methods for invasive weeds like Sahara mustard often involve a combination of prevention, early detection, and rapid response. This can include manual removal, herbicide application, biological control, and cultural practices that promote native vegetation. However, the specific methods and their effectiveness may vary depending on the region, habitat, and other factors.\n\nFurther research is needed to determine the most effective methods to control Sahara mustard in the Midwest USA region. It is recommended to consult with local experts, conduct field studies, and review relevant literature to develop a comprehensive control strategy for this invasive species."
+  }
+]

examples-work-out/example_results_20260302_202159.json

@@ -0,0 +1,20 @@
+[
+  {
+    "species": "Lycorma delicatula",
+    "region": "Midwest USA",
+    "model": "Llama-3 70B",
+    "mode": "Researcher",
+    "question": "How can I identify and report spotted lanternfly in Iowa?",
+    "availability": "**Common Name:** Unknown\n\nInformation for **Lycorma delicatula** is available in region(s): **Midwest USA**.",
+    "answer": "Based on the provided context from the Midwest USA region, here's the information to help you identify and report spotted lanternfly in Iowa:\n\n**Identification:**\n\n* Egg stage: Look for egg masses present in Iowa from late July to the following spring.\n* Nymphs: Identify by their striking colors - young nymphs are black with white spots, while older nymphs are red and black with white spots. They are often found on host trees, usually on branches, and will move quickly to avoid being seen.\n* Adults: Recognize by their pinkish-gray color with black spots, long grayish-black legs, and a second pair of red, white, and black wings beneath the gray spotted wings.\n\n**Reporting:**\n\nWhile there is no specific information in the provided context on how to report spotted lanternfly in Iowa, it is essential to report any sightings to the Iowa Department of Agriculture and Land Stewardship or local authorities to help track the spread of this invasive species.\n\n**Additional information:**\n\nFrom the context, we know that spotted lanternfly feed on a wide variety of plants, but prefer tree of heaven (Ailanthus altissima). This information can be useful when monitoring for the species.\n\n**General knowledge:**\n\nIt is crucial to note that early detection and reporting are critical in managing the spread of invasive species like the spotted lanternfly. If you suspect you have found a spotted lanternfly, take clear, close-up photos of the insect and the egg masses, and report the location and any other relevant details to the authorities.\n\nNo additional information from other regions (USA, Africa, or India) is relevant to this specific question."
+  },
+  {
+    "species": "Brassica tournefortii",
+    "region": "Midwest USA",
+    "model": "Llama-3 70B",
+    "mode": "Researcher",
+    "question": "What mechanical, chemical, and cultural control methods are recommended for managing Sahara mustard infestations?",
+    "availability": "**Common Name:** Unknown\n\nInformation for **Brassica tournefortii** is available in region(s): **Midwest USA**.",
+    "answer": "Based on the provided context, I must inform you that there is no specific information available in the Midwest USA region regarding mechanical, chemical, and cultural control methods for managing Sahara mustard infestations.\n\nHowever, I can provide some general knowledge on control methods for invasive weeds like Sahara mustard. Please note that these are not specific to the Midwest USA region and may require adaptation and testing for local conditions.\n\n**Mechanical Control:**\n\n* Hand-pulling or digging can be effective for small infestations, especially when the soil is moist. It is essential to remove the entire root system to prevent regrowth.\n* Mowing or cutting can be used to reduce seed production, but it may not completely eliminate the infestation.\n\n**Chemical Control:**\n\n* Herbicides can be used to control Sahara mustard, but the specific products and application rates may vary depending on the region and local regulations. It is crucial to choose a herbicide that is selective for the target species and minimizes harm to non-target plants and the environment.\n\n**Cultural Control:**\n\n* Preventing seed dispersal is critical in managing Sahara mustard infestations. This can be achieved by avoiding the movement of contaminated soil, equipment, and vehicles.\n* Restoring native vegetation can help outcompete Sahara mustard and reduce its spread.\n* Altering land use practices, such as changing grazing patterns or implementing conservation tillage, may also help reduce the spread of Sahara mustard.\n\nIt is essential to note that a comprehensive management plan should consider a combination of these control methods, as well as monitoring and early detection, to effectively manage Sahara mustard infestations.\n\nIf you are looking for region-specific information, I recommend consulting with local experts or conducting further research to develop a management plan tailored to the Midwest USA region."
+  }
+]

examples-work-out/example_results_20260302_202402.json

@@ -0,0 +1,20 @@
+[
+  {
+    "species": "Lycorma delicatula",
+    "region": "Midwest USA",
+    "model": "Llama-3 70B",
+    "mode": "Researcher",
+    "question": "How can I identify and report spotted lanternfly in Iowa?",
+    "availability": "**Common Name:** Unknown\n\nInformation for **Lycorma delicatula** is available in region(s): **Midwest USA**.",
+    "answer": "Based on the provided context from the Midwest USA region, here's the information to help you identify and report spotted lanternfly in Iowa:\n\n**Identification:**\n\n* Egg stage: Spotted lanternfly spend the winter in the egg stage, and egg masses would be present in Iowa from late July to the following spring.\n* Nymphs: Young nymphs are black with white spots, and older nymphs are red and black with white spots. They are fond of host trees, usually on branches, and will move quickly around the branch to avoid being seen.\n* Adults: Spotted lanternfly adults look like pinkish-gray cicadas or very large treehoppers. Their wings are pinkish-gray colored and covered in black spots. They have long legs that are grayish black. Beneath the gray spotted wings, there is a second pair of wings that are red, white, and black.\n\n**Reporting:**\n\n* According to the Iowa State University Yard and Garden Encyclopedia, you can report spotted lanternfly sightings in Iowa. The exact reporting process is not specified in the provided context, but it is recommended to contact the Iowa State University Extension or the Iowa Department of Agriculture and Land Stewardship for guidance on reporting procedures.\n\n**Additional Information:**\n\n* General knowledge: It is essential to report spotted lanternfly sightings promptly, as early detection and rapid response are critical in preventing the spread of this invasive species.\n\nNo additional information is available from other regions (USA, Africa, or India) that is specific to reporting spotted lanternfly in Iowa."
+  },
+  {
+    "species": "Brassica tournefortii",
+    "region": "Midwest USA",
+    "model": "Llama-3 70B",
+    "mode": "Researcher",
+    "question": "What mechanical, chemical, and cultural control methods are recommended for managing Sahara mustard infestations?",
+    "availability": "**Common Name:** Unknown\n\nInformation for **Brassica tournefortii** is available in region(s): **Midwest USA**.",
+    "answer": "Based on the provided context from the Midwest USA region, here are the recommended control methods for managing Sahara mustard infestations:\n\n**Mechanical Control (Midwest USA):** Monitoring the area for rodent caches and treating them accordingly is important. This suggests that mechanical control methods, such as hand-pulling or digging up, may be effective in managing Sahara mustard infestations, especially in areas where rodent caching is a significant means of spread.\n\n**Chemical Control (Midwest USA):** Early applications of chemicals may control Sahara mustard due to its extremely early development. Applications should be particularly effective if started before native species have begun to develop. However, it is noted that finding an agent that will control Sahara mustard without damaging canola crops may be challenging.\n\n**Cultural Control (No information available in Midwest USA context):** Unfortunately, there is no information available in the provided context from the Midwest USA region regarding cultural control methods for managing Sahara mustard infestations.\n\n**General Knowledge:** In general, cultural control methods for managing invasive species like Sahara mustard may include practices such as altering land use, modifying soil conditions, or using cover crops to outcompete the invasive species. However, more research is needed to determine the effectiveness of these methods for Sahara mustard specifically.\n\nNo relevant information was found in the other regions (USA, Africa, and India) that could supplement the information from the Midwest USA region."
+  }
+]

examples-work-out/example_results_20260303_104758.json

@@ -0,0 +1,56 @@
+[
+  {
+    "species": "Lycorma delicatula",
+    "region": "Midwest USA",
+    "model": "Llama-3 70B",
+    "mode": "Researcher",
+    "question": "How can I identify and report spotted lanternfly in Iowa?",
+    "availability": "**Common Name:** Spotted Lanternfly\n\nInformation for **Lycorma delicatula** is available in region(s): **Midwest USA, USA**.",
+    "answer": "Based on the provided context, here's the information on identifying and reporting spotted lanternfly in Iowa:\n\n**From Midwest USA:**\n\nTo identify spotted lanternfly in Iowa, you can look for the following characteristics:\n\n* Egg masses: Present in Iowa from late July to the following spring.\n* Nymphs: Black with white spots (young) or red and black with white spots (older). They are fond of host trees, usually on branches, and will move quickly to avoid being seen. Nymphs start hatching from eggs in April and develop throughout the early summer.\n* Adults: Pinkish-gray cicada-like or large treehopper-like, with pinkish-gray wings covered in black spots, grayish-black legs, and a second pair of red, white, and black wings beneath. Adults are present in Iowa from mid-July through a killing freeze.\n\n**Reporting:**\n\nAccording to the Iowa State University Yard and Garden Encyclopedia, you can report spotted lanternfly sightings to the authorities. The exact reporting process is not specified in the provided context, but it is likely that you can contact the Iowa Department of Agriculture and Land Stewardship or the Iowa State University Extension and Outreach office to report any sightings.\n\n**General Knowledge:**\nIt is essential to report spotted lanternfly sightings promptly, as early detection and rapid response are critical in managing the spread of this invasive species.\n\nPlease note that the provided context does not specify the exact reporting process or contact information. You may need to search for additional resources or contact local authorities for more information on reporting spotted lanternfly in Iowa."
+  },
+  {
+    "species": "Striga asiatica",
+    "region": "Africa",
+    "model": "Llama-3 70B",
+    "mode": "Researcher",
+    "question": "What is the most effective way to manage Striga in my maize field?",
+    "availability": "**Common Name:** Witch weed\n\nInformation for **Striga asiatica** is available in region(s): **Africa, USA**.",
+    "answer": "Based on the context from Africa, the most effective way to manage Striga in your maize field is to use a combination of integrated Striga management (ISM) technologies. According to Document 1 from Africa, this can include:\n\n1. Using Striga-resistant maize varieties as the foundation.\n2. Implementing maize-legume rotation to deplete the Striga seedbank.\n3. Adding legume intercropping (push-pull) to trigger suicidal germination.\n4. Applying nitrogen fertilizer to suppress strigolactone exudation.\n5. Deploying Fusarium oxysporum biocontrol where feasible.\n\nIt is emphasized that no single method suffices, and combining these complementary components is necessary for maximum effect.\n\nAdditionally, from the USA context, it is recommended to implement integrated management strategies combining cultural, chemical, and biological controls for effective long-term management of Striga asiatica (Document 1 from USA). This may include cultural control practices such as crop rotation, trap crops, deep tillage, and sanitation (Document 3 from USA).\n\nIt is also worth noting that monitoring and early detection are crucial for managing Striga infestations, and regular field inspections can help identify early signs of infestation (Document 1 from USA).\n\nIn general, it is essential to adopt a holistic approach that considers the specific conditions of your maize field and the local Striga population to develop an effective management strategy.\n\nPlease note that the provided context does not mention the specific efficacy of each method or the optimal combination of methods for your specific region or field conditions. Further research and experimentation may be necessary to determine the most effective management strategy for your specific situation."
+  },
+  {
+    "species": "Striga asiatica",
+    "region": "USA",
+    "model": "Llama-3 70B",
+    "mode": "Researcher",
+    "question": "What is the most effective way to manage Striga in my maize field?",
+    "availability": "**Common Name:** Witch weed\n\nInformation for **Striga asiatica** is available in region(s): **Africa, USA**.",
+    "answer": "Based on the provided context, here's a comprehensive answer to your question:\n\n**From the USA region:**\n\nFor effective management of Striga asiatica in your maize field, an integrated approach combining cultural, chemical, and biological controls is recommended (Document 1 from USA). This multi-faceted strategy can provide long-term management of Striga infestations.\n\nCultural control practices can play a crucial role in managing Striga. Implementing crop rotation with non-host crops can help reduce the Striga seed bank in the soil (Document 3 from USA). Additionally, planting trap crops that stimulate Striga germination but do not support its growth can help deplete the seed bank. Deep tillage can also be effective in burying Striga seeds deeper in the soil, reducing their viability. Furthermore, ensuring that farm equipment is clean can prevent the spread of Striga seeds between fields.\n\nChemical control options, such as pre-emergent and post-emergent herbicides, can also be used to control Striga infestations. Herbicides like imazapyr and glyphosate have been effective when used appropriately (Document 2 from USA).\n\n**From other regions:**\n\nWhile there is no specific information from the USA region on biological control options, research from Africa suggests that Fusarium oxysporum biocontrol can be an effective component of an integrated management strategy (Document 1 from Africa).\n\n**General knowledge:**\n\nIt's essential to note that monitoring and early detection are crucial for managing Striga infestations. Regular field inspections can help identify early signs of infestation, allowing for prompt action to be taken.\n\nIn summary, a comprehensive approach combining cultural, chemical, and biological controls, along with regular monitoring and early detection, is likely to be the most effective way to manage Striga in your maize field."
+  },
+  {
+    "species": "Spodoptera frugiperda",
+    "region": "India",
+    "model": "Llama-3 70B",
+    "mode": "Researcher",
+    "question": "What are the recommended IPM strategies for managing fall armyworm in my corn field?",
+    "availability": "**Common Name:** Fall Armyworm\n\nInformation for **Spodoptera frugiperda** is available in region(s): **India, Midwest USA, USA**.",
+    "answer": "Based on the provided context, here are the recommended IPM strategies for managing fall armyworm in your corn field in India:\n\n**From India:**\n\n1. **Cultural Control:** Implement crop rotation, intercropping, and planting shortly after major rainstorms to reduce pest incidence.\n2. **Monitoring:** Use pheromone traps (8 traps/40 lures per acre) for early detection and adult population monitoring, applied based on Economic Threshold Level (ETL).\n3. **Biological Control:** Apply Metarhizium anisopliae (entomopathogenic fungus) at 1 kg/acre.\n4. **Botanical/Biorational:** Use azadirachtin 1500 ppm (neem-based) at 1 L/acre.\n5. **Chemical Control (need-based):** Apply emamectin benzoate 5% SG at 100 g/acre, based on ETL, noting that resistance has developed to older insecticides like cypermethrin and lambda-cyhalothrin.\n\n**From other regions:**\n\n**From USA:**\n\n1. **Cultural Control/Management Practices:** Rotate crops to disrupt the lifecycle of the pest, adjust planting dates to avoid peak periods of armyworm activity, and practice field sanitation by removing crop residues and weeds that can serve as hosts.\n2. **Biological Control Options/Natural Enemies:** Conserve natural enemies like parasitoids (e.g., Trichogramma spp.) and predators (e.g., lady beetles and lacewings) through habitat management to help control fall armyworm populations.\n\n**General knowledge:** It is essential to adopt a holistic IPM approach, combining multiple strategies to manage fall armyworm effectively. Regular monitoring and threshold-based applications are crucial to minimize the development of resistance to control measures.\n\nPlease note that the recommended strategies may vary depending on the specific region, climate, and crop conditions. It is essential to consult with local experts and"
+  },
+  {
+    "species": "Spodoptera frugiperda",
+    "region": "USA",
+    "model": "Llama-3 70B",
+    "mode": "Researcher",
+    "question": "What are the recommended IPM strategies for managing fall armyworm in my corn field?",
+    "availability": "**Common Name:** Fall Armyworm\n\nInformation for **Spodoptera frugiperda** is available in region(s): **India, Midwest USA, USA**.",
+    "answer": "Based on the provided context, here are the recommended IPM strategies for managing fall armyworm in your corn field in the USA:\n\n**From USA:**\n\n1. **Chemical Control Options**: Insecticides may be used, but resistance management is critical due to the fall armyworm's ability to develop resistance. Use of insecticides should be based on monitoring and threshold levels to minimize unnecessary applications. Bacillus thuringiensis (Bt) products can be effective against larvae.\n2. **Biological Control Options/Natural Enemies**: Natural enemies include parasitoids such as Trichogramma spp. and predators like lady beetles and lacewings. Conservation of these natural enemies through habitat management can help control fall armyworm populations.\n3. **Cultural Control/Management Practices**:\n\t* **Crop Rotation**: Rotate crops to disrupt the lifecycle of the pest.\n\t* **Planting Date Adjustment**: Alter planting dates to avoid peak periods of armyworm activity.\n\t* **Field Sanitation**: Remove crop residues and weeds that can serve as hosts.\n\t* **Trap Crops**: Use trap crops to attract and manage populations away from main crops.\n\n**From Midwest USA:**\n\n1. **Scouting and Thresholds**: Begin scouting in mid-June and continue until silks dry in corn. Check whorls for larvae prior to tasseling in corn. Use a sweep net in soybean and alfalfa. Areas appearing drought stressed in alfalfa may indicate fall armyworm presence. Treat corn with foliar insecticides when 25% of plants are infested, caterpillars are small (<1 inch), and still exposed.\n2. **Management**: Consider managing grassy weeds, both in and near soybean and corn fields, to reduce armyworm problems. Natural enemies contribute to population reduction, especially in alfalfa. Transgenic Bt corn exhibits action against fall armyworm; however, some Bt proteins such as Cry1F and Vip3A"
+  },
+  {
+    "species": "Brassica tournefortii",
+    "region": "Midwest USA",
+    "model": "Llama-3 70B",
+    "mode": "Researcher",
+    "question": "What mechanical, chemical, and cultural control methods are recommended for managing Sahara mustard infestations?",
+    "availability": "**Common Name:** Sahara Mustard\n\nInformation for **Brassica tournefortii** is available in region(s): **Midwest USA, USA**.",
+    "answer": "Based on the provided context, here are the recommended mechanical, chemical, and cultural control methods for managing Sahara mustard infestations:\n\n**Mechanical Control (Midwest USA):**\nMonitoring the area for rodent caches and treating them accordingly is important. No other mechanical control methods are mentioned in the Midwest USA context.\n\n**Chemical Control (Midwest USA):**\nEarly applications of chemicals may control Sahara mustard due to its extremely early development. Applications should be particularly effective if started before native species have begun to develop.\n\n**Cultural Control (Midwest USA):**\nNo cultural control methods are mentioned in the Midwest USA context.\n\n**Additional Information from Other Regions:**\n\n**Cultural Control (USA):**\nManual removal is effective for small infestations, ensuring the entire root is removed to prevent regrowth. Mowing before seed set can reduce seed production but may need to be repeated due to staggered germination. Restoration of native vegetation can help outcompete Sahara Mustard.\n\n**Chemical Control (USA):**\nHerbicides such as glyphosate and 2,4-D can be effective when applied to young plants before they set seed. Pre-emergent herbicides can be used to prevent seed germination in heavily infested areas.\n\n**General Knowledge:**\nIt is essential to note that the effectiveness of these control methods may vary depending on the specific infestation site, soil type, and climate. A combination of methods may be necessary for optimal results. Additionally, prevention and early detection are crucial in managing Sahara mustard infestations, as they can prevent the establishment of a seed bank in the soil.\n\nIn summary, a comprehensive approach to managing Sahara mustard infestations may involve a combination of mechanical, chemical, and cultural control methods, including monitoring for rodent caches, early chemical applications, manual removal, mowing, and restoration of native vegetation."
+  }
+]

examples-work-out/species-verification.md

@@ -4,47 +4,61 @@ The 4 app regions are: **Midwest USA**, **USA**, **India**, **Africa**.
 
 Each vignette maps to one row in the motivation table and demonstrates a specific PestIDBot capability. The user question should naturally trigger that capability.
 
+**Database summary** (rebuilt 2026-03-03): **1410 total chunks** — Midwest USA: 388, USA: 913, India: 39, Africa: 70.
+
+**Latest comprehensive example run**: `example_results_20260303_110427.json` (6 examples covering all 4 vignettes with cross-regional demos, using updated prompt — no "Document N" citations, no general knowledge filler)
+
 ---
 
 ## 1. Spotted Lanternfly (*Lycorma delicatula*)
 
 **Table highlight**: Life-stage variation (eggs, instars, adults); risk of misidentification → **Life-stage robust ID; abstains when uncertain (Stage 1)**
 
-**Region**: Midwest USA
+**Stage 1 reference**: InsectNet Figure 2 in Chiranjeevi et al. (2025) shows life-stage classification results.
 
-**User question**: "I found this insect on my grapevine — can you identify it?" *(uploads photo showing a specific life stage, e.g. late instar or egg mass)*
+**Stage 2 example (AgLLM)**:
+- **Region**: Midwest USA (available in DB)
+- **Source PDF**: ISU Yard and Garden Extension — Spotted Lanternfly (3 pages: life cycle, damage, reporting)
+- **User question**: "How can I identify and report spotted lanternfly in Iowa?"
+- **What PestIDBot does**: Returns life-stage descriptions (eggs, nymphs with color variations, adults) from ISU extension data, plus Iowa-specific reporting guidance.
+- **Example output**: `example_results_20260303_110427.json` (Example 1)
 
-**What PestIDBot does**: Correctly identifies the life stage despite visual variation across stages. If uncertain, abstains and asks for re-imaging.
-
-**Database needs**: Life-stage descriptions/images for SLF (egg mass, early nymph, late nymph, adult), reporting and quarantine info for the selected US state/region.
+**Database status**:
+- Midwest USA: ✅ 6+ chunks from ISU PDF (3 pages ingested)
+- USA: ✅ GPT-4o generated IPM info (5 chunks)
 
 ---
 
-## 2. Witchweed (*Striga* spp.)
+## 2. Witchweed (*Striga asiatica*)
 
 **Table highlight**: Africa/India (smallholder constraints) → resistant varieties, intercrops; US → eradication, quarantine → **Region-filtered recommendations (Stage 2)**
 
-**Region**: Africa (primary), with cross-regional supplement from USA/India
-
-**User question**: "What is the most effective way to manage Striga in my maize field?" *(region: Africa)*
+**Stage 2 examples (AgLLM)**:
+- **Regions**: Africa (primary), USA (cross-regional)
+- **User question**: "What is the most effective way to manage Striga in my maize field?"
+- **What PestIDBot does**: For Africa, returns smallholder-appropriate IPM (resistant varieties, *Desmodium* push-pull, *Fusarium oxysporum* biocontrol, nitrogen fertilizer). For USA, leads with integrated cultural/chemical/biological approach (crop rotation, trap crops, imazapyr/glyphosate herbicides), supplements with African biocontrol research.
+- **Example output**: `example_results_20260303_110427.json` (Examples 2-3: Africa vs USA)
 
-**What PestIDBot does**: Returns African smallholder-appropriate IPM (resistant varieties, *Desmodium* push-pull, sanitation) first, then cross-regional supplements with transferability notes. Does NOT lead with US eradication protocols since those don't apply to the user's context.
-
-**Database needs**: African Striga IPM strategies (varieties, intercropping, sanitation, seed treatment). US eradication/quarantine info and India management info for cross-regional context.
+**Database status**:
+- Africa: ✅
+- USA: ✅
 
 ---
 
-## 3. Old World Bollworm (*Helicoverpa armigera*) [replaced with fall army worom with same regular aware guidance which had more regions. ]
-
-**Table highlight**: Management approved in US may not be approved in African/Indian systems → **Regulatory-aware guidance (Stage 2)**
+## 3. Fall Armyworm (*Spodoptera frugiperda*) [replaced Old World Bollworm — more regions available]
 
-**Region**: Africa (to show the contrast — US chemicals are NOT recommended because they're unavailable/unapproved)
+**Table highlight**: India: biorationals, ETL monitoring, insecticide resistance awareness; USA: Bt crops, synthetic insecticides → **Resistance-aware, resource-appropriate guidance (Stage 2)**
 
-**User question**: "How should I control bollworm in my cotton?" *(region: Africa)*
+**Stage 2 examples (AgLLM)**:
+- **Regions**: India (primary), USA (cross-regional), Midwest USA
+- **User question**: "What are the recommended IPM strategies for managing fall armyworm in my corn field?"
+- **What PestIDBot does**: For India, foregrounds ETL-based pheromone monitoring, biorationals (neem, *Metarhizium*), and emamectin benzoate with resistance warnings. For USA, leads with Bt crops, resistance management, mechanized scouting with ISU thresholds (25% infestation).
+- **Example output**: `example_results_20260303_110427.json` (Examples 4-5: India vs USA)
 
-**What PestIDBot does**: Provides region-appropriate controls (cultural practices, pheromone traps, biocontrol) instead of US-approved chemicals. Explicitly notes regulatory differences.
-
-**Database needs**: African bollworm management practices (cultural, biological, pheromone-based). US-approved chemicals for *H. armigera* (to know what NOT to recommend). Regulatory context on why they differ.
+**Database status**:
+- India: ✅
+- Midwest USA: ✅
+- USA: ✅
 
 ---
 
@@ -52,10 +66,15 @@ Each vignette maps to one row in the motivation table and demonstrates a specifi
 
 **Table highlight**: Port-of-entry screening; partial/low-light images; misidentification historically enabled spread → **Abstention → triage → escalate (Stage 1+2)**
 
-**Region**: USA (California)
-
-**User question**: "Can you identify this plant from my field survey?" *(uploads blurry/partial photo)*
+**Stage 1 reference**: WeedNet OOD detection and conformal prediction for abstention on poor images (Shen et al. 2025).
 
-**What PestIDBot does**: Abstains (low confidence on poor image), provides triage steps: re-imaging instructions, specimen collection protocol, diagnostic lab contact, reporting portal. Does NOT guess.
+**Stage 2 example (AgLLM)**:
+- **Region**: Midwest USA (available in DB), USA (cross-regional)
+- **Source PDF**: University of Nevada Extension — Identifying and Managing Sahara Mustard (3 pages: ID, habitat/impact, management methods)
+- **User question**: "What mechanical, chemical, and cultural control methods are recommended for managing Sahara mustard infestations?"
+- **What PestIDBot does**: Returns Midwest USA data (early chemical applications, rodent cache monitoring) plus USA cross-regional supplements (manual removal, mowing before seed set, glyphosate/2,4-D herbicides, native vegetation restoration).
+- **Example output**: `example_results_20260303_110427.json` (Example 6)
 
-**Database needs**: Specimen collection and chain-of-custody protocol, California diagnostic lab contacts, CDFA reporting portals, re-imaging best practices.
+**Database status**:
+- Midwest USA: ✅ 11+ chunks from extension PDF (3 pages ingested)
+- USA: ✅ GPT-4o generated IPM info

retrieval_evaluation_results.json

@@ -1,60 +1,60 @@
 {
   "no_filter": {
     "precision@1": {
-      "mean": 0.64,
-      "std": 0.48,
+      "mean": 0.5,
+      "std": 0.5,
       "count": 100
     },
     "precision@2": {
-      "mean": 0.76,
-      "std": 0.4270831300812524,
+      "mean": 0.72,
+      "std": 0.4489988864128729,
       "count": 100
     },
     "precision@4": {
-      "mean": 0.81,
-      "std": 0.39230090491866054,
+      "mean": 0.79,
+      "std": 0.40730823708832603,
       "count": 100
     },
     "precision@8": {
-      "mean": 0.85,
-      "std": 0.3570714214271425,
+      "mean": 0.87,
+      "std": 0.33630343441600474,
       "count": 100
     },
     "ndcg@1": {
-      "mean": 0.64,
-      "std": 0.48,
+      "mean": 0.5,
+      "std": 0.5,
       "count": 100
     },
     "ndcg@2": {
-      "mean": 0.7157115704285749,
-      "std": 0.41895779074707734,
+      "mean": 0.6388045457857207,
+      "std": 0.4236799147359738,
       "count": 100
     },
     "ndcg@4": {
-      "mean": 0.7400183360093088,
-      "std": 0.38986711559166154,
+      "mean": 0.6724180769471885,
+      "std": 0.38940915964484785,
       "count": 100
     },
     "ndcg@8": {
-      "mean": 0.754344341157148,
-      "std": 0.3684202134419992,
+      "mean": 0.7001274694006319,
+      "std": 0.35105364601238853,
       "count": 100
     }
   },
   "species_only": {
     "precision@1": {
-      "mean": 0.68,
-      "std": 0.466476151587624,
+      "mean": 0.69,
+      "std": 0.46249324319388707,
       "count": 100
     },
     "precision@2": {
-      "mean": 0.84,
-      "std": 0.3666060555964672,
+      "mean": 0.85,
+      "std": 0.3570714214271425,
       "count": 100
     },
     "precision@4": {
-      "mean": 0.93,
-      "std": 0.25514701644346144,
+      "mean": 0.97,
+      "std": 0.17058722109231983,
       "count": 100
     },
     "precision@8": {
@@ -63,35 +63,35 @@
       "count": 100
     },
     "ndcg@1": {
-      "mean": 0.68,
-      "std": 0.466476151587624,
+      "mean": 0.69,
+      "std": 0.46249324319388707,
       "count": 100
     },
     "ndcg@2": {
-      "mean": 0.7809487605714333,
-      "std": 0.36580132584863706,
+      "mean": 0.7909487605714333,
+      "std": 0.3578989169880122,
       "count": 100
     },
     "ndcg@4": {
-      "mean": 0.824562291732901,
-      "std": 0.29154459287790513,
+      "mean": 0.8467893540558369,
+      "std": 0.2505360340326592,
       "count": 100
     },
     "ndcg@8": {
-      "mean": 0.8508003372990416,
-      "std": 0.22561628946474954,
+      "mean": 0.8575532873325957,
+      "std": 0.21977117994271048,
       "count": 100
     }
   },
   "region_only": {
     "precision@1": {
-      "mean": 0.69,
-      "std": 0.462493243193887,
+      "mean": 0.53,
+      "std": 0.4990991885387111,
       "count": 100
     },
     "precision@2": {
-      "mean": 0.78,
-      "std": 0.4142463035441596,
+      "mean": 0.77,
+      "std": 0.4208325082500163,
       "count": 100
     },
     "precision@4": {
@@ -100,40 +100,40 @@
       "count": 100
     },
     "precision@8": {
-      "mean": 0.87,
-      "std": 0.33630343441600474,
+      "mean": 0.89,
+      "std": 0.31288975694324034,
       "count": 100
     },
     "ndcg@1": {
-      "mean": 0.69,
-      "std": 0.462493243193887,
+      "mean": 0.53,
+      "std": 0.4990991885387111,
       "count": 100
     },
     "ndcg@2": {
-      "mean": 0.7467836778214312,
-      "std": 0.4100495706548825,
+      "mean": 0.6814231408571497,
+      "std": 0.4014970336757105,
       "count": 100
     },
     "ndcg@4": {
-      "mean": 0.7747039745636328,
-      "std": 0.37245769233603554,
+      "mean": 0.7150366720186176,
+      "std": 0.36120571502682,
       "count": 100
     },
     "ndcg@8": {
-      "mean": 0.7857743640416571,
-      "std": 0.35433577663047267,
+      "mean": 0.7327460644720609,
+      "std": 0.33335934745276274,
       "count": 100
     }
   },
   "species_and_region": {
     "precision@1": {
-      "mean": 0.79,
-      "std": 0.40730823708832603,
+      "mean": 0.74,
+      "std": 0.4386342439892262,
       "count": 100
     },
     "precision@2": {
-      "mean": 0.91,
-      "std": 0.2861817604250837,
+      "mean": 0.89,
+      "std": 0.31288975694324034,
       "count": 100
     },
     "precision@4": {
@@ -147,23 +147,23 @@
       "count": 100
     },
     "ndcg@1": {
-      "mean": 0.79,
-      "std": 0.40730823708832603,
+      "mean": 0.74,
+      "std": 0.4386342439892262,
       "count": 100
     },
     "ndcg@2": {
-      "mean": 0.8657115704285749,
-      "std": 0.2971736181074952,
+      "mean": 0.8346394630357188,
+      "std": 0.32107291980904945,
       "count": 100
     },
     "ndcg@4": {
-      "mean": 0.9036318671707767,
-      "std": 0.19819838555518796,
+      "mean": 0.8832529941971865,
+      "std": 0.2080496269798261,
       "count": 100
     },
     "ndcg@8": {
-      "mean": 0.907500395243122,
-      "std": 0.18377359707383636,
+      "mean": 0.8871215222695319,
+      "std": 0.19476255984038815,
       "count": 100
     }
   }

species-organized/PestID Species - Organized.xlsx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c2c04042374df7b1c0122979a109e36553d9d7f3c560aa8e5a6cd258eca23066
-size 63579
+oid sha256:965f330dfbc74381e712a8ea4e65633ed5a07fcae68c69dae797c1b533fdcc6b
+size 98454

species-organized/species_statistics.txt

@@ -4,23 +4,23 @@ PEST SPECIES ANALYSIS - STATISTICS SUMMARY
 
 1. OVERALL SPECIES COUNTS
 ----------------------------------------
-Total species: 126
+Total species: 128
 
 By Region:
-  Midwest USA:  80 species ( 63.5%)
-  Africa    :  35 species ( 27.8%)
-  India     :  11 species (  8.7%)
+  Midwest USA:  82 species ( 64.1%)
+  Africa    :  35 species ( 27.3%)
+  India     :  11 species (  8.6%)
 
 2. INSECT VS WEED DISTRIBUTION
 ----------------------------------------
 Overall:
-  Insects: 65 (51.6%)
-  Weeds:   61 (48.4%)
+  Insects: 66 (51.6%)
+  Weeds:   62 (48.4%)
 
 By Region:
   Midwest USA:
-    Insects: 44
-    Weeds:   36
+    Insects: 45
+    Weeds:   37
   Africa:
     Insects: 10
     Weeds:   25
@@ -36,7 +36,7 @@ Midwest USA:
   Std Dev:   11.74%
   Min:       40.00%
   Max:      100.00%
-  Missing:  19 (23.8%)
+  Missing:  21 (25.6%)
 
 Africa:
   Mean:      89.81%
@@ -62,10 +62,10 @@ Overall (all regions):
 
 4. IPM INFORMATION COVERAGE
 ----------------------------------------
-Midwest USA:  0/80 species (  0.0%)
+Midwest USA:  0/82 species (  0.0%)
 Africa    : 35/35 species (100.0%)
 India     : 11/11 species (100.0%)
-Overall   : 46/126 species ( 36.5%)
+Overall   : 46/128 species ( 35.9%)
 
 5. TOP 10 SPECIES BY ACCURACY
 ----------------------------------------

species-organized/species_table.tex

@@ -82,6 +82,7 @@ Midwest USA & \textit{Pirate bugs} & Pirate bugs & insect & — & No \\
 Midwest USA & \textit{Soldier beetles} & Soldier beetles & insect & — & No \\
 Midwest USA & \textit{Podisus maculiventris} & Spined soldier bug & insect & — & No \\
 Midwest USA & \textit{Tachinid flies} & Tachinid flies & insect & — & No \\
+Midwest USA & \textit{Lycorma delicatula} & Spotted Lanternfly & insect & — & No \\
 Midwest USA & \textit{Empoasca fabae} & potato leafhopper & insect & 97.0 & No \\
 Midwest USA & \textit{Striacosta albicosta} & western bean cutworm & insect & 97.0 & No \\
 Midwest USA & \textit{Hypena scabra} & green cloverworm & insect & 96.0 & No \\
@@ -115,6 +116,7 @@ Midwest USA & \textit{Solanum ptycanthum} & Eastern black nightshade & weed &
 Midwest USA & \textit{Conyza canadensis} & Horseweed & weed & — & No \\
 Midwest USA & \textit{Kochia scoparia} & Kochia & weed & — & No \\
 Midwest USA & \textit{Sinapis arvensis} & Wild mustard & weed & — & No \\
+Midwest USA & \textit{Brassica tournefortii} & Sahara Mustard & weed & — & No \\
 Midwest USA & \textit{Ambrosia artemisiifolia} & common Ragweed & weed & 95.0 & No \\
 Midwest USA & \textit{Stellaria media} & commonChickweed & weed & 95.0 & No \\
 Midwest USA & \textit{Equisetum arvense} & Field Horsetail & weed & 95.0 & No \\

usa_ipm_checkpoint.json

@@ -115,7 +115,237 @@
     "Cleome spinosa",
     "Medicago falcata",
     "Spodoptera frugiperda",
-    "Striga asiatica"
+    "Striga asiatica",
+    "Papaipema nebris",
+    "Helicoverpa zea",
+    "Lolium multiflorum",
+    "Lady beetles",
+    "Lygus lineolaris",
+    "Agrotis ipsilon",
+    "Vanessa cardui",
+    "Hypena scabra",
+    "Sitona hispidulus",
+    "Echinochloa crus-galli",
+    "Popillia japonica",
+    "Kochia scoparia",
+    "Tetranychus urticae",
+    "Lamium amplexicaule",
+    "Chaetocnema pulicaria",
+    "Chrysodeixis includens",
+    "Lacewings",
+    "Mythimna unipuncta",
+    "Cyperus esculentus",
+    "Setaria viridis",
+    "Setaria faberi",
+    "Hypera postica",
+    "Hydraecia immanis",
+    "Stellaria media",
+    "Colias eurytheme",
+    "Parasitoid wasps",
+    "Ground beetles",
+    "Microtechnites bractatus",
+    "Cerotoma trifurcata",
+    "Podisus maculiventris",
+    "Digitaria sanguinalis",
+    "Conyza canadensis",
+    "Loxostege cereralis",
+    "Equisetum arvense",
+    "Dectes texanus",
+    "Spodoptera ornithogalli",
+    "Stenolophus lecontei",
+    "Hypera zoilus",
+    "Trianthema triquetrum",
+    "Flower fly larvae",
+    "Aphis glycines",
+    "Ambrosia artemisiifolia",
+    "Agromyza frontella",
+    "Choristoneura parallela",
+    "Damsel bugs",
+    "Persicaria pensylvanica",
+    "Philaenus spumarius",
+    "Eleusine indica",
+    "Drosophila suzukii",
+    "Lactuca serriola",
+    "Abutilon theophrasti",
+    "Spodoptera frugiperda",
+    "Xanthium strumarium",
+    "Amaranthus palmeri",
+    "Brassica tournefortii",
+    "Delia platura",
+    "Setaria pumila",
+    "Calomycterus setarius",
+    "Ostrinia nubilalis",
+    "Amaranthus retroflexus",
+    "Helicoverpa armigera",
+    "Medicago minima",
+    "Sida spinosa",
+    "Chenopodium album",
+    "Halyomorpha halys",
+    "Soldier beetles",
+    "Solanum ptycanthum",
+    "Mirabilis nyctaginea",
+    "Equisetum hyemale",
+    "Amaranthus tuberculatus",
+    "Cleome rutidosperma",
+    "Aphis craccivora",
+    "Helianthus annuus",
+    "Sorghum bicolor",
+    "Cyperus difformis",
+    "Lycorma delicatula",
+    "Striacosta albicosta",
+    "Empoasca fabae",
+    "Cleome spinosa",
+    "Datura stramonium",
+    "Cleome viscosa",
+    "Striga asiatica",
+    "Pirate bugs",
+    "Amaranthus viridis",
+    "Ambrosia trifida",
+    "Sinapis arvensis",
+    "Bromus tectorum",
+    "Cyperus mindorensis",
+    "Euborellia annulipes",
+    "Odontota horni",
+    "Cyperus prolifer",
+    "Cleome aculeata",
+    "Medicago polymorpha",
+    "Cleome monophylla",
+    "Cyperus brevifolius",
+    "Daucus carota",
+    "Capsella bursa-pastoris",
+    "Euphorbia dentata",
+    "Icerya purchasi",
+    "Cyperus rotundus",
+    "Aphis spiraecola",
+    "Cyperus entrerianus",
+    "Salsola tragus",
+    "Trianthema portulacastrum",
+    "Spodoptera litura",
+    "Cyperus iria l.",
+    "Cleome houtteana",
+    "Cleome gynandra",
+    "Medicago lupulina",
+    "Cyperus haspan",
+    "Atherigona reversura",
+    "Tachinid flies",
+    "Nezara viridula",
+    "Medicago falcata",
+    "Resseliella maxima",
+    "Cyperus mindorensis",
+    "Spodoptera litura",
+    "Trianthema triquetrum",
+    "Amaranthus viridis",
+    "Vanessa cardui",
+    "Colias eurytheme",
+    "Daucus carota",
+    "Brassica tournefortii",
+    "Euphorbia dentata",
+    "Loxostege cereralis",
+    "Podisus maculiventris",
+    "Ground beetles",
+    "Abutilon theophrasti",
+    "Cyperus haspan",
+    "Odontota horni",
+    "Stellaria media",
+    "Sida spinosa",
+    "Cyperus brevifolius",
+    "Setaria pumila",
+    "Xanthium strumarium",
+    "Capsella bursa-pastoris",
+    "Persicaria pensylvanica",
+    "Digitaria sanguinalis",
+    "Chenopodium album",
+    "Lamium amplexicaule",
+    "Icerya purchasi",
+    "Calomycterus setarius",
+    "Aphis spiraecola",
+    "Ambrosia artemisiifolia",
+    "Halyomorpha halys",
+    "Solanum ptycanthum",
+    "Cyperus difformis",
+    "Flower fly larvae",
+    "Datura stramonium",
+    "Echinochloa crus-galli",
+    "Kochia scoparia",
+    "Lady beetles",
+    "Delia platura",
+    "Choristoneura parallela",
+    "Aphis craccivora",
+    "Hypera postica",
+    "Lygus lineolaris",
+    "Dectes texanus",
+    "Sinapis arvensis",
+    "Spodoptera frugiperda",
+    "Setaria viridis",
+    "Pirate bugs",
+    "Lacewings",
+    "Ostrinia nubilalis",
+    "Salsola tragus",
+    "Equisetum hyemale",
+    "Agromyza frontella",
+    "Damsel bugs",
+    "Agrotis ipsilon",
+    "Resseliella maxima",
+    "Cyperus prolifer",
+    "Helianthus annuus",
+    "Cleome houtteana",
+    "Lolium multiflorum",
+    "Microtechnites bractatus",
+    "Mythimna unipuncta",
+    "Sitona hispidulus",
+    "Parasitoid wasps",
+    "Eleusine indica",
+    "Soldier beetles",
+    "Striacosta albicosta",
+    "Tachinid flies",
+    "Ambrosia trifida",
+    "Atherigona reversura",
+    "Cleome monophylla",
+    "Cyperus entrerianus",
+    "Bromus tectorum",
+    "Cyperus iria l.",
+    "Conyza canadensis",
+    "Nezara viridula",
+    "Cyperus esculentus",
+    "Equisetum arvense",
+    "Mirabilis nyctaginea",
+    "Lactuca serriola",
+    "Hypera zoilus",
+    "Philaenus spumarius",
+    "Helicoverpa armigera",
+    "Aphis glycines",
+    "Amaranthus palmeri",
+    "Lycorma delicatula",
+    "Setaria faberi",
+    "Amaranthus tuberculatus",
+    "Cerotoma trifurcata",
+    "Spodoptera ornithogalli",
+    "Empoasca fabae",
+    "Euborellia annulipes",
+    "Amaranthus retroflexus",
+    "Chaetocnema pulicaria",
+    "Drosophila suzukii",
+    "Helicoverpa zea",
+    "Popillia japonica",
+    "Hypena scabra",
+    "Stenolophus lecontei",
+    "Cyperus rotundus",
+    "Tetranychus urticae",
+    "Hydraecia immanis",
+    "Papaipema nebris",
+    "Cleome spinosa",
+    "Sorghum bicolor",
+    "Chrysodeixis includens",
+    "Cleome rutidosperma",
+    "Medicago lupulina",
+    "Trianthema portulacastrum",
+    "Cleome gynandra",
+    "Cleome aculeata",
+    "Medicago minima",
+    "Medicago falcata",
+    "Striga asiatica",
+    "Cleome viscosa",
+    "Medicago polymorpha"
   ],
-  "last_run": "2026-02-04T21:47:41.950969"
+  "last_run": "2026-03-03T10:44:42.836240"
 }

vector-databases-deployed/db5-agllm-data-isu-field-insects-all-species/{27ac9297-abc2-406e-8919-7670a60055f1 → ba4b6a9d-5012-4456-8635-7ea7d6cb7a71}/data_level0.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bb76ae0f6ca830a9a048b0cf53962a78c88c5c7fcda63fc846077d3456eb3890
+oid sha256:0df39a7224e80c717058f5c4e95a21faf300fe9102de146ce32a9f3e2e7d5a76
 size 62840000

vector-databases-deployed/db5-agllm-data-isu-field-insects-all-species/{27ac9297-abc2-406e-8919-7670a60055f1 → ba4b6a9d-5012-4456-8635-7ea7d6cb7a71}/header.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ec666c9828420c69fc6b597461d8c18487becec1527c7d1cff9b898cbb393c2d
+oid sha256:f23645792d8664b1d497ca854c4459f7c29d757472fdb6591d327f0303495f93
 size 100

vector-databases-deployed/db5-agllm-data-isu-field-insects-all-species/ba4b6a9d-5012-4456-8635-7ea7d6cb7a71/index_metadata.pickle

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd94800fffc4e1b3aca003062b6cf6e8eb3c4683b77ed4eae5f2d378cd41d183
+size 129852

vector-databases-deployed/db5-agllm-data-isu-field-insects-all-species/{27ac9297-abc2-406e-8919-7670a60055f1 → ba4b6a9d-5012-4456-8635-7ea7d6cb7a71}/length.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0f2fe12bccc27d05f521a0a00eb43912172a7113283d0a99b45af0b2da569582
+oid sha256:e7e2dcff542de95352682dc186432e98f0188084896773f1973276b0577d5305
 size 40000

vector-databases-deployed/db5-agllm-data-isu-field-insects-all-species/ba4b6a9d-5012-4456-8635-7ea7d6cb7a71/link_lists.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38d6d79326a1813390ce0d391cfa648cb550eba165548e079e7dd1f6ea240ef6
+size 12508

vector-databases-deployed/db5-agllm-data-isu-field-insects-all-species/chroma.sqlite3

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:08dce2645f4aa422368f3fc4e975e58a518875b335458e83f60a0205c46f27b8
-size 10715136
+oid sha256:a8ba30aefc5e3dbf2fee25c2fe8f49eda6fe5396d31f8d8dbcc34660ed926392
+size 17203200