Spaces:

---

CGIAR

/

ifpri-dp-rag

like 2

Running

Proposal: Add a CGSpace collection ingestion pipeline

Hi,

I would like to propose a contribution to this Space by adding an optional ingestion script for CGSpace collections.

While testing the project locally, I adapted the ingestion workflow so that the application can build a new RAG index from a specific public CGSpace collection, instead of only relying on the current IFPRI Discussion Papers dataset.

What the proposed script does

The proposed ingest_cgspace_collection.py script:

• retrieves public items from a CGSpace collection using its collection UUID;

• uses the CGSpace REST API and Discovery endpoint;

• extracts basic item metadata such as title, authors, year, type, abstract, and handle URL;

• finds available PDF bitstreams;

• downloads public PDFs;

• extracts text page by page;

• splits the text into chunks;

• generates embeddings using the configured embedding model;

• builds a FAISS index;

• generates the files already expected by the current app.py:

  • faiss_index/
  • metadata.json
  • output-items.csv
  • output-pdfs.csv

Example usage

python ingest_cgspace_collection.py --collection-uuid COLLECTION_UUID --limit 10