Update README.md
Browse files
README.md
CHANGED
|
@@ -16,7 +16,7 @@ Introducing Petrichor, a new Game Master helper for TTRPGs. Petrichor is a dual-
|
|
| 16 |
|
| 17 |
# Data Overview
|
| 18 |
|
| 19 |
-
The curated static corpus consists of 32 Project Gutenberg novels, 8 TTRPG rulebooks or handbooks, and 5 TTRPG scenario books. The novels act as a pure and classic form of narrative creativity, while TTRPG related books depict more active and actionable narrative structures. The corpus is comprised of a wide variety of file types (PDF, txt, docx, markdown, etc.), which are either public domain, licensed under Creative Commons, or fall under other permissive licensing.
|
| 20 |
|
| 21 |
### Corpus and Campaign Notes Addition
|
| 22 |
|
|
@@ -28,7 +28,7 @@ Campaign notes may be optionally supplied to the system, which should ideally be
|
|
| 28 |
|
| 29 |
### Data Evaluation Process
|
| 30 |
|
| 31 |
-
Due to the nature of the creative corpus, a standard test split was unsuitable for the task. Instead, three benchmarks were evaluated, two utilizing open-ended prompting, while
|
| 32 |
|
| 33 |
---
|
| 34 |
|
|
|
|
| 16 |
|
| 17 |
# Data Overview
|
| 18 |
|
| 19 |
+
The curated static corpus consists of 32 Project Gutenberg novels, 8 TTRPG rulebooks or handbooks, and 5 TTRPG scenario books. The novels act as a pure and classic form of narrative creativity, while TTRPG related books depict more active and actionable narrative structures. The corpus is comprised of a wide variety of file types (PDF, txt, docx, markdown, etc.), which are either public domain, licensed under Creative Commons, or fall under other permissive licensing. Other than using different packages to pull text for different file types, the files themselves were not altered and all file types were treated as equally as possible.
|
| 20 |
|
| 21 |
### Corpus and Campaign Notes Addition
|
| 22 |
|
|
|
|
| 28 |
|
| 29 |
### Data Evaluation Process
|
| 30 |
|
| 31 |
+
Due to the nature of the creative corpus, a standard test split was unsuitable for the task. Instead, three benchmarks were evaluated, two utilizing open-ended prompting, one is domain-specific and created by hand and the other a subset of an existing database of generic creative prompts, while the final benchmark utilized a synthetic dataset created from the corpus to test competence of retrieval. These benchmarks collectively serve the purpose of a "holdout" set. All benchmarks were evaluated by LLM-as-judge and are described in more depth in the **Evaluation** section.
|
| 32 |
|
| 33 |
---
|
| 34 |
|