Keep retrieval, synthesis and chat consistent; correct the corpus counts

app.py: the reranker's relevance scores now travel into the synthesis and chat
prompts, the chat is handed the synthesis the user is reading so follow-ups
build on it instead of contradicting it, and a shared evidence_note states
plainly when nothing cleared the weak-match threshold.

README.md: the corpus is Perseus plus First1KGreek, 364 authors and 1,644
works, not the 100 authors and 813 texts previously advertised; the Scaife
links open the English translation alongside where one exists.

README.md

@@ -7,22 +7,22 @@ sdk: gradio
 sdk_version: 6.19.0
 app_file: app.py
 pinned: false
-short_description: Semantic search over the Perseus Greek corpus
+short_description: Semantic search over Perseus and First1KGreek
 ---
 
 # Zetema
 
-Semantic search over the [Perseus Digital Library](http://www.perseus.tufts.edu/) Greek corpus using local open-weight language models. Ask questions in English (or Greek) and find which ancient authors write about a given topic.
+Semantic search over the [Perseus Digital Library](http://www.perseus.tufts.edu/) and [First1KGreek](https://github.com/OpenGreekAndLatin/First1K-Greek) Greek corpora using local open-weight language models. Ask questions in English (or Greek) and find which ancient authors write about a given topic.
 
 ## What it does
 
 - **Cross-lingual retrieval**: type a query in English and retrieve matching passages in ancient Greek — the embedding model maps both into the same semantic space
 - **Author ranking**: see at a glance which authors discuss a topic most
-- **Read in context**: every result links to the passage in the [Scaife Viewer](https://scaife.perseus.org) via its CTS URN, so you can read it in its full context
+- **Read in context**: every result links to the passage in the [Scaife Viewer](https://scaife.perseus.org) via its CTS URN, with the English translation opened side by side when the corpus has one
 - **Reranked results**: dense retrieval fetches a candidate pool, then a cross-encoder reranker (`Qwen/Qwen3-Reranker-0.6B`) orders the final results
 - **Century and genre filters**: narrow the search to particular centuries or genres (history, tragedy, philosophy, …), both derived from the TLG Canon; result cards show each author's dates
 - **LLM synthesis**: optionally send top passages to a local LLM (Gemma 3 by default, via Hugging Face `transformers`, streamed) to get a summary in English with key phrase translations
-- **100 authors, 813 Greek texts**: Plato, Aristotle, Thucydides, Homer, Aeschylus, Sophocles, Euripides, Herodotus, and many more
+- **364 authors, 1,644 Greek works**: Perseus and [First1KGreek](https://github.com/OpenGreekAndLatin/First1K-Greek) together — Plato, Aristotle, Thucydides, Homer, Aeschylus, Sophocles, Euripides, Herodotus, and many more
 
 ## Requirements
 
@@ -47,14 +47,15 @@ Install a `torch` build that matches your CUDA driver — a mismatched build sil
 
 ### 2. Point to the corpus
 
-The `perseus` symlink in the project root must point to your local copy of the Perseus XML corpus. Adjust as needed:
+Two symlinks in the project root point at your local corpus checkouts. Both are optional individually — `ingest.py` skips a missing one — but the index only covers what it can find:
 
 ```bash
-# example — change the target to match your actual path
+# example — change the targets to match your actual paths
 ln -sf /path/to/perseus/xml perseus
+ln -sf /path/to/First1KGreek/data first1k
 ```
 
-The corpus should contain directories named `tlg0001`, `tlg0003`, … each with `__cts__.xml` metadata and TEI XML text files.
+Each corpus should contain directories named `tlg0001`, `tlg0003`, … each with `__cts__.xml` metadata and TEI XML text files. Where the two corpora carry the same work, Perseus takes precedence and the duplicate is skipped.
 
 ### 3. Build the index (one-time, ~10–20 min on GPU, longer on CPU)
 
@@ -158,19 +159,23 @@ Query (English or Greek)
 
 ```
 zetema/
-├── ingest.py              # corpus ingestion: parse → embed → store
-├── app.py                 # Gradio web UI
-├── extract_tlg_dates.py   # derives data/tlg_dates.json from the TLG Canon
+├── ingest.py                    # corpus ingestion: parse → embed → store
+├── app.py                       # Gradio web UI
+├── extract_tlg_dates.py         # derives data/tlg_dates.json from the TLG Canon
+├── extract_eng_translations.py  # derives data/eng_translations.json from the corpora
 ├── requirements.txt
-├── perseus -> ...         # symlink to Perseus XML corpus
+├── perseus -> ...               # symlink to Perseus XML corpus
+├── first1k -> ...               # symlink to First1KGreek XML corpus
 └── data/
-    ├── tlg_dates.json     # author dates + genres (checked in)
-    └── perseus_rag_qwen.duckdb # built by ingest.py
+    ├── tlg_dates.json           # author dates + genres (checked in)
+    ├── eng_translations.json    # work → English version label (checked in)
+    └── perseus_rag_qwen.duckdb  # built by ingest.py
 ```
 
 ## References
 
 - [Perseus Digital Library](http://www.perseus.tufts.edu/), Tufts University
+- [First1KGreek](https://github.com/OpenGreekAndLatin/First1K-Greek), Open Greek and Latin
 - [Thesaurus Linguae Graecae](https://www.tlg.uci.edu/) Canon (author dates and genres)
 - [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) embedding model
 - [Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B) reranker

app.py

@@ -430,13 +430,38 @@ CONTEXT_PASSAGES = 8  # how many retrieved passages the LLM sees
 
 
 def passages_context(passages: list[dict]) -> str:
-    """Format retrieved passages as numbered context for LLM prompts."""
+    """Format retrieved passages as numbered context for LLM prompts.
+
+    Carries the reranker's relevance score with each passage. Without it the
+    synthesis and chat treat a 0.99 match and a 0.31 near-miss as equally
+    authoritative, discarding the ranking the retrieval path just computed.
+    """
     return "\n\n".join(
-        f"[{i}] {p['author']}, {p['work']} §{p['section_ref']}\n{p['text'][:600]}"
+        f"[{i}] {p['author']}, {p['work']} §{p['section_ref']} "
+        f"(relevance {p['score']:.2f})\n{p['text'][:600]}"
         for i, p in enumerate(passages[:CONTEXT_PASSAGES], 1)
     )
 
 
+def evidence_note(passages: list[dict]) -> str:
+    """Shared preamble telling the LLM how to read the relevance scores."""
+    best = max((p["score"] for p in passages[:CONTEXT_PASSAGES]), default=0.0)
+    note = (
+        "Passages are listed best-first, and each carries the reranker's "
+        "relevance score from 0 to 1. Weight them accordingly: a passage "
+        f"scoring below about {WEAK_MATCH_THRESHOLD:.2f} is a weak match that "
+        "may not bear on the query at all."
+    )
+    if best < WEAK_MATCH_THRESHOLD:
+        note += (
+            " Note that NO passage here scored above "
+            f"{WEAK_MATCH_THRESHOLD:.2f}, so the corpus probably does not "
+            "address this query. Say so plainly rather than assembling an "
+            "answer out of near-misses."
+        )
+    return note
+
+
 def synthesize_stream(query: str, passages: list[dict], model_name: str):
     """Stream an LLM synthesis of the retrieved passages."""
     context = passages_context(passages)
@@ -445,6 +470,7 @@ def synthesize_stream(query: str, passages: list[dict], model_name: str):
         f"You are a scholar of ancient Greek literature. "
         f"The following passages from the Perseus corpus were retrieved for the query:\n"
         f"  \"{query}\"\n\n"
+        f"{evidence_note(passages)}\n\n"
         f"Passages (in ancient Greek):\n{context}\n\n"
         f"Please:\n"
         f"1. Identify the main themes or concepts related to the query found in these passages.\n"
@@ -456,12 +482,26 @@ def synthesize_stream(query: str, passages: list[dict], model_name: str):
     yield from llm_chat_stream([{"role": "user", "content": prompt}], model_name)
 
 
-def chat_system_prompt(query: str, passages: list[dict]) -> str:
-    return (
+def chat_system_prompt(query: str, passages: list[dict],
+                       synthesis: str = "") -> str:
+    prompt = (
         "You are a scholar of ancient Greek literature helping a user explore "
         "passages retrieved from the Perseus corpus for the search query "
         f"\"{query}\".\n\n"
+        f"{evidence_note(passages)}\n\n"
         f"Retrieved passages (in ancient Greek):\n{passages_context(passages)}\n\n"
+    )
+    if synthesis.strip():
+        # The user is reading this summary on screen, so the chat has to build
+        # on it. Without it the chat answers as though the summary never
+        # existed and can flatly contradict what the user just read.
+        prompt += (
+            "You have already given the user this summary of these passages:\n"
+            f"---\n{synthesis.strip()}\n---\n"
+            "Stay consistent with it. If you now think part of it was wrong, "
+            "say so explicitly rather than quietly changing your account.\n\n"
+        )
+    return prompt + (
         "Answer the user's questions about these passages: interpret, translate, "
         "compare, and point to the relevant authors and works. Refer to passages "
         "by their bracketed numbers. If a question goes beyond what the passages "
@@ -745,7 +785,7 @@ def run_search(query, top_k, author_filter, century_filter, genre_filter,
                 yield html, top_authors, acc, results, query, []
 
 
-def chat_respond(message, history, llm_model, query, results):
+def chat_respond(message, history, llm_model, query, results, synthesis):
     """Chat about the last search's passages; streams the assistant reply."""
     message = (message or "").strip()
     if not message:
@@ -761,7 +801,8 @@ def chat_respond(message, history, llm_model, query, results):
         return
 
     messages = [
-        {"role": "system", "content": chat_system_prompt(query, results)}
+        {"role": "system",
+         "content": chat_system_prompt(query, results, synthesis or "")}
     ] + history
     history.append({"role": "assistant", "content": ""})
     for acc in llm_chat_stream(messages, llm_model):
@@ -858,7 +899,10 @@ def build_ui() -> gr.Blocks:
         search_btn.click(fn=run_search, inputs=search_inputs, outputs=search_outputs)
         query_box.submit(fn=run_search, inputs=search_inputs, outputs=search_outputs)
 
-        chat_inputs = [chat_input, chatbot, llm_model, query_state, results_state]
+        # synthesis_md is passed by value: the chat needs to see the summary
+        # the user is currently reading.
+        chat_inputs = [chat_input, chatbot, llm_model, query_state,
+                       results_state, synthesis_md]
         chat_send.click(fn=chat_respond, inputs=chat_inputs, outputs=[chatbot, chat_input])
         chat_input.submit(fn=chat_respond, inputs=chat_inputs, outputs=[chatbot, chat_input])