Adding auto embedding

RAG_APP_README.md

@@ -0,0 +1,54 @@
+# Abalone RAG Chatbot
+
+This project implements a Retrieval-Augmented Generation (RAG) chatbot about Abalone using LangChain + OpenAI with a Streamlit frontend. It's designed to be deployed on Hugging Face Spaces.
+
+Contents
+- `app.py` - Streamlit app entrypoint
+- `src/ingest.py` - Ingest files from `data/` into a persisted Chroma vectorstore
+- `src/vectorstore.py` - Helpers to build/load the Chroma vectorstore and return a retriever
+- `src/qa_chain.py` - Build the conversational retrieval QA chain
+- `data/` - Put Abalone source files here (CSV/MD/TXT/PDF)
+- `vectorstore/` - Persisted vectorstore directory (created by ingestion)
+
+Quickstart (local)
+
+1. Create a venv and install dependencies:
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+
+2. Set your OpenAI API key:
+
+```bash
+export OPENAI_API_KEY="sk-..."
+```
+
+3. Add Abalone files into `data/` (for example `abalone.csv`).
+
+4. Build the vectorstore:
+
+```bash
+python -m src.ingest --data-dir ./data --persist-dir ./vectorstore
+```
+
+5. Run the Streamlit app:
+
+```bash
+streamlit run app.py
+```
+
+Deploying to Hugging Face Spaces
+
+- Add `OPENAI_API_KEY` in the Spaces secrets (Settings -> Secrets).
+- Push this repository to your HF Space. HF will install `requirements.txt` and run the Streamlit app.
+- On first run, click the "Ingest data" button or allow the app to rebuild the index.
+
+Security
+- Do NOT commit your OpenAI API key. Use HF Spaces Secrets for deployment.
+
+License
+- MIT
+

app.py

@@ -109,16 +109,32 @@ st.success("Vectorstore and retriever are ready.")
 
 chain = make_conversational_chain(retriever, model_name=model_name)
 
+def format_source_label(meta: dict, index: int) -> str:
+    source = (
+            meta.get("source")
+            or meta.get("file_path")
+            or meta.get("path")
+            or meta.get("document_id")
+            or "Unknown source"
+    )
+    return f"[{index}] {source}"
+
 if st.session_state["chat_history"]:
     st.subheader("Conversation")
     for i, turn in enumerate(st.session_state["chat_history"]):
         st.markdown(f"**You:** {turn['question']}")
         st.markdown(f"**Abalone Bot:** {turn['answer']}")
-        if turn.get("sources"):
-            with st.expander(f"Show sources for question {i + 1}"):
-                for j, src in enumerate(turn["sources"], start=1):
-                    st.markdown(f"**Source {j}:**")
-                    meta = src.get("metadata", {})
+        sources = turn.get("sources") or []
+        if sources:
+            st.markdown("**Sources:**")
+            for src in sources:
+                label = format_source_label(src.get("metadata", {}) or {}, src.get("index", 0))
+                st.markdown(f"- {label}")
+            with st.expander(f"Show source details for question {i + 1}"):
+                for src in sources:
+                    label = format_source_label(src.get("metadata", {}) or {}, src.get("index", 0))
+                    st.markdown(f"**{label}**")
+                    meta = src.get("metadata", {}) or {}
                     if meta:
                         st.write(meta)
                     preview = src.get("content_preview", "")
@@ -156,7 +172,7 @@ if send_clicked and user_input:
         source_docs = result.get("source_documents") or []
 
         sources_for_ui = []
-        for sd in source_docs:
+        for idx, sd in enumerate(source_docs, start=1):
             if isinstance(sd, dict):
                 meta = sd.get("metadata", {}) or {}
                 content_preview = sd.get("page_content") or sd.get("content") or sd.get("text", "")
@@ -169,6 +185,7 @@ if send_clicked and user_input:
                 content_preview = ""
             sources_for_ui.append(
                 {
+                    "index": idx,
                     "metadata": meta,
                     "content_preview": str(content_preview)[:500],
                 }
@@ -188,13 +205,20 @@ if send_clicked and user_input:
     st.markdown(f"**Abalone Bot:** {answer}")
 
     if sources_for_ui:
-        with st.expander("Show sources for this answer"):
-            for i, src in enumerate(sources_for_ui, start=1):
-                st.markdown(f"**Source {i}:**")
-                if src["metadata"]:
-                    st.write(src["metadata"])
-                if src["content_preview"]:
-                    st.write(src["content_preview"])
+        st.markdown("**Sources:**")
+        for src in sources_for_ui:
+            label = format_source_label(src.get("metadata", {}) or {}, src.get("index", 0))
+            st.markdown(f"- {label}")
+        with st.expander("Show source details for this answer"):
+            for src in sources_for_ui:
+                label = format_source_label(src.get("metadata", {}) or {}, src.get("index", 0))
+                st.markdown(f"**{label}**")
+                meta = src.get("metadata", {}) or {}
+                if meta:
+                    st.write(meta)
+                preview = src.get("content_preview", "")
+                if preview:
+                    st.write(preview)
 
 elif send_clicked and not user_input:
     st.warning("Please enter a question before clicking Send.")

data/abalone.txt

@@ -0,0 +1,165 @@
+Abalone
+Description
+Abalones are members of a large class (Gastropoda) of molluscs having one-piece shells. They belong to
+the family Haliotidae and the genus Haliotis, which means sea ear, referring to the flattened shape of the
+shell.
+Abalone shells are rounded or oval with a large dome towards one end. The shell has a row of respiratory
+pores. The muscular foot has strong suction power permitting the abalone to clamp tightly to rocky
+surfaces. An epipodium, a sensory structure and extension of the foot that bears tentacles, circles the foot
+and projects beyond the shell edge in the living abalone. Nine species of abalone occur in North America:
+black (H. cracherodii), flat (H. walallensis), green (H. fulgens), pink (H. corrugata), pinto (H.
+kamtschatkana), red (H. rufescens), threaded (H. assimilis), Western Atlantic (H. pourtalesii), and white
+(H. sorenseni) abalone.
+Black abalone (H. cracherodii) have black and smooth epipodium and tentacles. The shell surface is
+black or dark blue, and smooth. There are 5 to 9 open pores, and the pores are flush with the shell
+surface. Black abalone range from Mendocino County, California to southern Baja California. They are
+found in intertidal and shallow subtidal zones down to a depth of about 20 feet. Black abalone reach 7.75
+inches in length, but are commonly 5 to 6 inches long.
+Flat abalone (H. walallensis) have a mottled yellowish and brown epipodium, with a pebbly appearing
+surface and lacy edge. The tentacles are greenish and slender. The shell is flattened, narrow, and
+marked with low ribs. There are 5 to 6 open pores, and the pore edges are moderately elevated above
+the shell surface. Flat abalone range from British Columbia, Canada to San Diego, California. They are
+found in the subtidal zone from 20 feet down to at least 70 feet. Flat abalone reach 7 inches in length, but
+are commonly under 5 inches.
+Green abalone (H. fulgens) have a mottled cream and brown epipodium, with tubercles scattered on the
+surface and a frilly edge. The tentacles are olive green. The shell is usually brown, and its surface marked
+with many low, flat-topped ribs that run parallel to the pores. There are 5 to 7 open pores, and the pore
+edges are elevated above the shell surface. A groove often parallels the outer edge of the line of pores.
+Green abalone range from Point Conception, California to Bahia Magdalena, Baja California. They are
+found in the intertidal and subtidal zones down to at least 30 feet. Green abalone are often found in
+crevices where surfgrass and algal cover is dense. They reach 10 inches in length, but are generally
+smaller.
+Pink abalone (H. corrugata) have a mottled black and white epipodium with many tubercles on the
+surface and a lacy edge. The foot is yellow to light orange. The tentacles are black. The shell is thick and
+its surface is marked with wavy corrugations. There are 2 to 4 open pores, and pore edges are strongly
+elevated above the surface. Pink abalone range from Point Conception, California to Santa Maria Bay,
+Baja California. They are found in the subtidal zone from 20 feet down to at least 120 feet, commonly in
+beds of giant kelp. Pink abalone reach 10 inches in length, but individuals over 7 inches long are now
+rare.
+Pinto abalone (H. kamtschatkana) have a mottled pale yellow to dark brown epipodium, with a pebbly
+appearing surface and lacy edge. Tentacles are yellowish brown, or occasionally green, and thin. The
+shell is irregularly mottled and narrow. There are 3 to 6 open pores, and the pore edges are elevated
+above the shell surface. A groove often parallels the line of pores. Pinto abalone range from Sitka, Alaska
+to Monterey, California. They are found in the intertidal and subtidal zones down to at least 70 feet. Pinto
+abalone reach 6.49 inches in length, but are commonly 4 inches long. Pinto abalone are also known
+regionally as northern abalone.
+Red abalone (H. rufescens) usually have a black epipodium, but some specimens have a barred black
+and cream pattern on their epipodium. The surface of the epipodium is smooth and broadly scalloped
+along the edge. The area around the foot is black and the sole is tan to grey. The tentacles are black. The
+shell surface is generally brick red and the inside edge is often red. There are 3 to 4 open pores, and the
+pores are moderately elevated above the shell surface. Red abalone range from Sunset Bay, Oregon to
+Tortugas, Baja California. North of Point Conception, they are found in the intertidal and subtidal zones
+down to at least 60 feet. South of Point Conception, they are found in the subtidal zone down to over 100
+feet. Red abalone reach 12.3 inches in length, but are commonly 7 to 9 inches long.
+Threaded abalone (H. assimilis) have a mottled pale yellow to dark brown epipodium with a pebbly
+appearing surface and frilly edge. The tentacles are yellowish brown, short and thin. The shell is oval and
+the surface is marked with prominent ribs interspersed with narrow ones. There are 4 to 6 open pores,
+and the pores are moderately elevated above the shell surface. Threaded abalone range from San Luis
+Obispo County, California to Bahia Tortugas, Baja California. They are found in the subtidal zone from 20
+feet down to at least 80 feet, commonly on rock surfaces. Threaded abalone reach 6 inches in length, but
+are commonly smaller. Threaded abalone are considered a subspecies of the pinto abalone by some
+scientists.
+Western Atlantic abalone (H. pourtalesii) have a yellowish epipodium with large and small sensory
+tentacles. The sole of the foot is tan. The shell is reddish-orange. Western Atlantic abalone range from
+North Carolina through the Gulf of Mexico to Brazil. They are found from 187 feet down to at least 1,200
+feet on hard substrates. The largest recorded shell had a length of about 1.2 inches.
+White abalone (H. sorenseni) have a tan and pebbly epipodium. The sole of the foot is orange. The shell
+is deep, thin and oval. There are 3 to 5 open pores, and the edges of the pores are elevated above the
+shell surface. White abalone range from Point Conception to Bahia Tortugas, Baja California. Most white
+abalone are found in the Channel Islands in California. White abalone are found in the subtidal zone
+down to at least 200 feet. They are commonly found in open, exposed areas. White abalone reach 10
+inches in length, but are commonly 5 to 8 inches long.
+Natural History
+Abalones reach sexual maturity at a small size, and fertility is high and increases exponentially with size.
+Sexes are separate and fertilization is external. The eggs and sperm broadcast into the water through the
+pores with the respiratory current. A 1.5 inch abalone may spawn 10,000 eggs or more at a time, while an
+8 inch abalone may spawn 11 million or more eggs. The spawning season varies among species with
+black, green and pink abalone spawning between spring and fall, and pinto abalone spawning during the
+summer. Red abalone in some locations spawn throughout the year. The fertilized eggs hatch into
+floating larvae that feed on plankton until their shells begin to form. Once the shell forms, the juvenile
+abalone sinks to the bottom where it clings to rocks and crevices with its single powerful foot. Settling
+rates appear to be variable. After settling, abalones change their diet and feed on macroalgae.
+Except for black abalone, hybridization for abalone species is not uncommon in areas where several
+species occur together. There are 12 recognized hybrids in southern California and northern Baja
+California.
+Limited growth information is available for abalones. Commercial sizes of 6.25 inches for pinks, seven
+inches for greens and 7.75 inches for reds are reached after a minimum of 10 to 15 years in southern
+California. Pinto abalone reach about 2.5 inches in a minimum of 6 years.
+Juvenile abalones feed on rock-encrusting coralline algae and on diatom and bacterial films. Adult
+abalones feed primarily on loose pieces of marine algae drifting with the surge or current. Large brown
+algae such as giant kelp, bull kelp, feather boa kelp and elk kelp are preferred, although other species of
+algae may be eaten at various times.
+Abalone eggs and larvae are consumed by filter-feeding fish and shellfish. Predators of juvenile abalones
+include crabs, lobsters, gastropods, octopuses, seastars, and fishes. The bat ray in southern California
+and the sea otter in central California prey selectively on larger abalones.
+Production
+In decreasing order of total catch between 1950 and 1995, red (46.6%), pink (41.2%), black (8.7%), green
+(3.5%), and white (>1%) abalones have all been harvested in California. Since 1993, only red abalone
+have been taken commercially, and the Fish and Game Commission closed all red abalone harvest south
+of San Francisco in May 1997. Pinto abalone are commercially harvested in Alaska and British Columbia.
+Flat and threaded abalones have limited distributions and neither is common. The western Atlantic
+abalone is rare and is not fished commercially.
+Aquaculture of red, pink, and green abalones occurs in California. There is limited aquaculture of green
+and H. diversicolor supertexta abalones in Hawaii.
+California. The commercial fishery for abalones in California began in the 1850's. Chinese Americans
+initially harvested intertidal green and black abalones with skiffs using long, hooked poles. This fishery
+was eliminated in California in 1900 by closure of shallow waters to commercial harvest. Japanese
+American divers followed the Chinese Americans as the fishery moved to the subtidal zone. Initially, free
+divers working from barrel floats harvested abalones. Later, hard-hat divers harvested abalones from
+deeper waters. In the late 1950's, "hooka" gear, which supplied air from the surface to divers using light
+masks, fins and wet suits, began replacing hard-hat gear. Since the 1970's, multi-hose hooka gear and
+specialized, high-speed, seaworthy boats have become common in the fishery.
+In California, abalone divers must use underwater diving gear consisting of an above-surface air pump
+operated from a boat and at least 100 feet of air hose, and must be fully submerged while taking abalone.
+Abalones may be taken only by hand or with abalone irons. An abalone iron is a flat device not more than
+36" long and not less than 1/16 inch thick, with rounded smooth edges and a curve with a radius of less
+than 18 inches. The commercial abalone fishery in California is managed through size limits, limits on the
+number of permits for commercial abalone divers, and restrictions on harvesting areas. Minimum
+commercial size limits in California are: 7-3/4 inches for red abalone, 7 inches for green abalone, 6-1/4
+inches for pink or white abalone, 5-3/4 inches for black abalone, and 4 inches for pinto, threaded, and flat
+abalone. Commercial harvesting is prohibited during January, February and August. A moratorium on
+commercial harvesting of black abalone began in July, 1993, and extends through January 1, 1997. It is
+unlikely that stocks of black abalone will recover enough for the fishery to reopen. In June, 1994, the
+California Department of Fish and Game proposed and the Fish and Game Commission adopted
+effective January 1, 1995 a two-year closure on sport and commercial harvesting of pink, green and white
+abalone. Prices to fishermen for red abalone were around $500 to $600 per dozen in 1993-94.
+The California commercial abalone harvest reached a record 5.4 million pounds in 1957. Since then,
+commercial harvests have declined dramatically to about 461,376 pounds in 1993. Current stocks of most
+abalone species in central and southern California are over utilized. This is the combined result of
+commercial harvest efficiency, increased market demand, sport fishery expansion, an expanding
+population of sea otters, pollution of mainland habitat, unexplained mortalities of black abalone due to a
+condition known as "withering syndrome," and loss of kelp populations associated with El Niño events.
+Management efforts through size limits and limits on commercial harvesting permits have been
+ineffective. Reseeding experiments have not been successful. Commercial abalone harvesting in
+California may be eliminated if the sea otter range is not contained. Studies in a California fishery reserve
+have shown that even protected populations cannot support a fishery within the sea otter range in central
+California. New laws pending in the 1997 Legislature would establish a multi-year moratorium on the
+commercial and recreational harvest of all species of abalone south of the entrance to San Francisco Bay
+until stocks have demonstrated some level of recovery and a new management plan is in effect.
+Alaska. The southeast Alaska commercial abalone fishery was sporadic and local prior to 1971. Shore
+picking was the primary harvesting method, but after 1960 some scuba gear was used. The fishery
+increased dramatically during the 1970's due to improved scuba gear, increased product demand, and
+the use of larger vessels. The Alaska abalone harvest reached a record 315,000 pounds in 1978-79, and
+then fell to about 36,000 pounds in 1992-93 when a minimum size limit was instituted. The Alaska pinto
+abalone fishery is managed through guideline harvest ranges, a minimum legal size of 3.75 inches, a
+restrictive season, and local area closures for conservation and food fisheries. The fishery opens in
+October to remain outside spawning and settling periods. Guideline harvests prior to 1988-89 varied
+33,000 to 57,000 pounds per year. The season was shortened each year, and in 1993-94 the most
+productive areas were closed after 6 days and a catch of 37,000 pounds.
+British Columbia. Prior to 1971, the British Columbia commercial pinto abalone fishery was sporadic
+and local. Shore picking was the main harvest method, but after 1960 some scuba gear was used. The
+fishery accelerated rapidly during the 1970's due to improved scuba gear, reduced access to herring and
+salmon fisheries, acceptance of the pinto abalone in the Japanese market, increased product demand,
+and the introduction of larger vessels with freezer capacity. Abalone landings peaked in 1977 at 474.8
+metric tons and then declined rapidly. The fishery was later closed to rebuild stocks.
+Products
+During the early years of the abalone fishery, abalones were dried and smoked, or canned for export, and
+sold fresh for local markets. Currently, most abalones are exported to Japan, either fresh or frozen whole.
+The U.S. market is primarily in California for live abalone for the sashimi market, and for some fresh and
+frozen steaks for restaurants.
+A major shift in U.S. marketing occurred after the black abalone moratorium in 1993. Red abalone became
+the primary export product. High prices increased the incentive for illegal harvesting in closed areas.
+Abalone steaks are produced by removing the foot, trimming, slicing, and tenderizing. Yield from a live
+abalone is roughly 15 percent. Shells are used for mother-of-pearl, souvenirs, and jewelry.
+References
+(References section preserved exactly as in the PDF.)

vectorstore/chroma-collections.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2f2f346a015c1ffec6f8a3c535ac4ea2a99fe14f441a424e373b42248ac0fbe
+size 601

vectorstore/chroma-embeddings.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f26b3ec8f49b9a9e03eebdc6185167fcb1a871609e75b9471817a8e0c5cdaf3
+size 208