Spaces:
Sleeping
Sleeping
Update methods_draft.md
Browse files- methods_draft.md +9 -12
methods_draft.md
CHANGED
|
@@ -1,6 +1,5 @@
|
|
| 1 |
# Methods Document
|
| 2 |
|
| 3 |
-
```markdown
|
| 4 |
# Concept Atlas: Methods & Application Guide
|
| 5 |
### German Curriculum Semantic Analysis — Technical & Pedagogical Documentation
|
| 6 |
|
|
@@ -98,7 +97,7 @@ Each row in `curriculum_excerpts.csv` represents one curriculum excerpt:
|
|
| 98 |
|
| 99 |
Before analysis, the corpus undergoes the following cleaning steps:
|
| 100 |
|
| 101 |
-
|
| 102 |
Raw CSV
|
| 103 |
→ Normalise column names (lowercase, underscores)
|
| 104 |
→ Fill missing values with empty strings
|
|
@@ -106,7 +105,7 @@ Raw CSV
|
|
| 106 |
→ Strip whitespace from text_excerpt and search_term
|
| 107 |
→ Remove excerpts shorter than 20 characters
|
| 108 |
→ Derive search_term_lower for case-insensitive concept matching
|
| 109 |
-
|
| 110 |
|
| 111 |
Concept subsets are built by **exact match** on `search_term_lower`, with
|
| 112 |
automatic fallback to **partial string match** if fewer than 10 exact matches
|
|
@@ -129,7 +128,7 @@ The full analysis pipeline runs sequentially in a single click. All
|
|
| 129 |
computationally expensive steps are cached to disk so that subsequent
|
| 130 |
exploration is instantaneous.
|
| 131 |
|
| 132 |
-
|
| 133 |
CSV ingestion
|
| 134 |
│
|
| 135 |
▼
|
|
@@ -163,7 +162,7 @@ Enriched parquet export
|
|
| 163 |
│
|
| 164 |
└──► data/enriched_corpus.csv
|
| 165 |
cache/enriched_corpus.parquet
|
| 166 |
-
|
| 167 |
|
| 168 |
### Caching strategy
|
| 169 |
|
|
@@ -281,7 +280,7 @@ Traditional methods (e.g. LDA) work on word co-occurrence statistics.
|
|
| 281 |
|
| 282 |
### Pipeline within BERTopic
|
| 283 |
|
| 284 |
-
|
| 285 |
Sentence embeddings (768-D)
|
| 286 |
│
|
| 287 |
▼
|
|
@@ -297,7 +296,7 @@ Sentence embeddings (768-D)
|
|
| 297 |
│
|
| 298 |
▼
|
| 299 |
Topic labels + per-document probabilities
|
| 300 |
-
|
| 301 |
|
| 302 |
### HDBSCAN: density-based clustering
|
| 303 |
|
|
@@ -594,7 +593,6 @@ automatically invalidated (different key → different filename → recomputed).
|
|
| 594 |
|
| 595 |
### Pushing to HuggingFace
|
| 596 |
|
| 597 |
-
```bash
|
| 598 |
# Authenticate
|
| 599 |
huggingface-cli login
|
| 600 |
|
|
@@ -604,14 +602,14 @@ huggingface-cli upload deirdosh/curriculum_analysis_german \
|
|
| 604 |
|
| 605 |
huggingface-cli upload deirdosh/curriculum_analysis_german \
|
| 606 |
./data data --repo-type=space
|
| 607 |
-
|
| 608 |
|
| 609 |
The `enriched_corpus.csv` adds BERTopic `topic_id` and UMAP coordinates
|
| 610 |
(`umap2_x`, `umap2_y`, `umap3_x`, `umap3_y`, `umap3_z`) to every excerpt,
|
| 611 |
making the enriched dataset independently useful for downstream analysis
|
| 612 |
without re-running the pipeline.
|
| 613 |
|
| 614 |
-
|
| 615 |
|
| 616 |
## 12. Educational Applications
|
| 617 |
|
|
@@ -739,7 +737,6 @@ in the same CSV format enables direct cross-national comparison.
|
|
| 739 |
|
| 740 |
To run the pipeline locally or contribute new analysis:
|
| 741 |
|
| 742 |
-
```bash
|
| 743 |
# Clone the Space
|
| 744 |
git clone https://huggingface.co/spaces/deirdosh/curriculum_analysis_german
|
| 745 |
cd curriculum_analysis_german
|
|
@@ -749,7 +746,7 @@ pip install -r requirements.txt
|
|
| 749 |
|
| 750 |
# Run locally
|
| 751 |
python app.py
|
| 752 |
-
|
| 753 |
|
| 754 |
A standard laptop (8 GB RAM, no GPU) can run the full pipeline in
|
| 755 |
approximately 15–20 minutes on first run. GPU acceleration reduces this to
|
|
|
|
| 1 |
# Methods Document
|
| 2 |
|
|
|
|
| 3 |
# Concept Atlas: Methods & Application Guide
|
| 4 |
### German Curriculum Semantic Analysis — Technical & Pedagogical Documentation
|
| 5 |
|
|
|
|
| 97 |
|
| 98 |
Before analysis, the corpus undergoes the following cleaning steps:
|
| 99 |
|
| 100 |
+
|
| 101 |
Raw CSV
|
| 102 |
→ Normalise column names (lowercase, underscores)
|
| 103 |
→ Fill missing values with empty strings
|
|
|
|
| 105 |
→ Strip whitespace from text_excerpt and search_term
|
| 106 |
→ Remove excerpts shorter than 20 characters
|
| 107 |
→ Derive search_term_lower for case-insensitive concept matching
|
| 108 |
+
|
| 109 |
|
| 110 |
Concept subsets are built by **exact match** on `search_term_lower`, with
|
| 111 |
automatic fallback to **partial string match** if fewer than 10 exact matches
|
|
|
|
| 128 |
computationally expensive steps are cached to disk so that subsequent
|
| 129 |
exploration is instantaneous.
|
| 130 |
|
| 131 |
+
|
| 132 |
CSV ingestion
|
| 133 |
│
|
| 134 |
▼
|
|
|
|
| 162 |
│
|
| 163 |
└──► data/enriched_corpus.csv
|
| 164 |
cache/enriched_corpus.parquet
|
| 165 |
+
|
| 166 |
|
| 167 |
### Caching strategy
|
| 168 |
|
|
|
|
| 280 |
|
| 281 |
### Pipeline within BERTopic
|
| 282 |
|
| 283 |
+
|
| 284 |
Sentence embeddings (768-D)
|
| 285 |
│
|
| 286 |
▼
|
|
|
|
| 296 |
│
|
| 297 |
▼
|
| 298 |
Topic labels + per-document probabilities
|
| 299 |
+
|
| 300 |
|
| 301 |
### HDBSCAN: density-based clustering
|
| 302 |
|
|
|
|
| 593 |
|
| 594 |
### Pushing to HuggingFace
|
| 595 |
|
|
|
|
| 596 |
# Authenticate
|
| 597 |
huggingface-cli login
|
| 598 |
|
|
|
|
| 602 |
|
| 603 |
huggingface-cli upload deirdosh/curriculum_analysis_german \
|
| 604 |
./data data --repo-type=space
|
| 605 |
+
|
| 606 |
|
| 607 |
The `enriched_corpus.csv` adds BERTopic `topic_id` and UMAP coordinates
|
| 608 |
(`umap2_x`, `umap2_y`, `umap3_x`, `umap3_y`, `umap3_z`) to every excerpt,
|
| 609 |
making the enriched dataset independently useful for downstream analysis
|
| 610 |
without re-running the pipeline.
|
| 611 |
|
| 612 |
+
|
| 613 |
|
| 614 |
## 12. Educational Applications
|
| 615 |
|
|
|
|
| 737 |
|
| 738 |
To run the pipeline locally or contribute new analysis:
|
| 739 |
|
|
|
|
| 740 |
# Clone the Space
|
| 741 |
git clone https://huggingface.co/spaces/deirdosh/curriculum_analysis_german
|
| 742 |
cd curriculum_analysis_german
|
|
|
|
| 746 |
|
| 747 |
# Run locally
|
| 748 |
python app.py
|
| 749 |
+
|
| 750 |
|
| 751 |
A standard laptop (8 GB RAM, no GPU) can run the full pipeline in
|
| 752 |
approximately 15–20 minutes on first run. GPU acceleration reduces this to
|