deirdosh commited on
Commit
46128be
·
verified ·
1 Parent(s): 2b36cf6

Update methods_draft.md

Browse files
Files changed (1) hide show
  1. methods_draft.md +9 -12
methods_draft.md CHANGED
@@ -1,6 +1,5 @@
1
  # Methods Document
2
 
3
- ```markdown
4
  # Concept Atlas: Methods & Application Guide
5
  ### German Curriculum Semantic Analysis — Technical & Pedagogical Documentation
6
 
@@ -98,7 +97,7 @@ Each row in `curriculum_excerpts.csv` represents one curriculum excerpt:
98
 
99
  Before analysis, the corpus undergoes the following cleaning steps:
100
 
101
- ```
102
  Raw CSV
103
  → Normalise column names (lowercase, underscores)
104
  → Fill missing values with empty strings
@@ -106,7 +105,7 @@ Raw CSV
106
  → Strip whitespace from text_excerpt and search_term
107
  → Remove excerpts shorter than 20 characters
108
  → Derive search_term_lower for case-insensitive concept matching
109
- ```
110
 
111
  Concept subsets are built by **exact match** on `search_term_lower`, with
112
  automatic fallback to **partial string match** if fewer than 10 exact matches
@@ -129,7 +128,7 @@ The full analysis pipeline runs sequentially in a single click. All
129
  computationally expensive steps are cached to disk so that subsequent
130
  exploration is instantaneous.
131
 
132
- ```
133
  CSV ingestion
134
 
135
 
@@ -163,7 +162,7 @@ Enriched parquet export
163
 
164
  └──► data/enriched_corpus.csv
165
  cache/enriched_corpus.parquet
166
- ```
167
 
168
  ### Caching strategy
169
 
@@ -281,7 +280,7 @@ Traditional methods (e.g. LDA) work on word co-occurrence statistics.
281
 
282
  ### Pipeline within BERTopic
283
 
284
- ```
285
  Sentence embeddings (768-D)
286
 
287
 
@@ -297,7 +296,7 @@ Sentence embeddings (768-D)
297
 
298
 
299
  Topic labels + per-document probabilities
300
- ```
301
 
302
  ### HDBSCAN: density-based clustering
303
 
@@ -594,7 +593,6 @@ automatically invalidated (different key → different filename → recomputed).
594
 
595
  ### Pushing to HuggingFace
596
 
597
- ```bash
598
  # Authenticate
599
  huggingface-cli login
600
 
@@ -604,14 +602,14 @@ huggingface-cli upload deirdosh/curriculum_analysis_german \
604
 
605
  huggingface-cli upload deirdosh/curriculum_analysis_german \
606
  ./data data --repo-type=space
607
- ```
608
 
609
  The `enriched_corpus.csv` adds BERTopic `topic_id` and UMAP coordinates
610
  (`umap2_x`, `umap2_y`, `umap3_x`, `umap3_y`, `umap3_z`) to every excerpt,
611
  making the enriched dataset independently useful for downstream analysis
612
  without re-running the pipeline.
613
 
614
- ---
615
 
616
  ## 12. Educational Applications
617
 
@@ -739,7 +737,6 @@ in the same CSV format enables direct cross-national comparison.
739
 
740
  To run the pipeline locally or contribute new analysis:
741
 
742
- ```bash
743
  # Clone the Space
744
  git clone https://huggingface.co/spaces/deirdosh/curriculum_analysis_german
745
  cd curriculum_analysis_german
@@ -749,7 +746,7 @@ pip install -r requirements.txt
749
 
750
  # Run locally
751
  python app.py
752
- ```
753
 
754
  A standard laptop (8 GB RAM, no GPU) can run the full pipeline in
755
  approximately 15–20 minutes on first run. GPU acceleration reduces this to
 
1
  # Methods Document
2
 
 
3
  # Concept Atlas: Methods & Application Guide
4
  ### German Curriculum Semantic Analysis — Technical & Pedagogical Documentation
5
 
 
97
 
98
  Before analysis, the corpus undergoes the following cleaning steps:
99
 
100
+
101
  Raw CSV
102
  → Normalise column names (lowercase, underscores)
103
  → Fill missing values with empty strings
 
105
  → Strip whitespace from text_excerpt and search_term
106
  → Remove excerpts shorter than 20 characters
107
  → Derive search_term_lower for case-insensitive concept matching
108
+
109
 
110
  Concept subsets are built by **exact match** on `search_term_lower`, with
111
  automatic fallback to **partial string match** if fewer than 10 exact matches
 
128
  computationally expensive steps are cached to disk so that subsequent
129
  exploration is instantaneous.
130
 
131
+
132
  CSV ingestion
133
 
134
 
 
162
 
163
  └──► data/enriched_corpus.csv
164
  cache/enriched_corpus.parquet
165
+
166
 
167
  ### Caching strategy
168
 
 
280
 
281
  ### Pipeline within BERTopic
282
 
283
+
284
  Sentence embeddings (768-D)
285
 
286
 
 
296
 
297
 
298
  Topic labels + per-document probabilities
299
+
300
 
301
  ### HDBSCAN: density-based clustering
302
 
 
593
 
594
  ### Pushing to HuggingFace
595
 
 
596
  # Authenticate
597
  huggingface-cli login
598
 
 
602
 
603
  huggingface-cli upload deirdosh/curriculum_analysis_german \
604
  ./data data --repo-type=space
605
+
606
 
607
  The `enriched_corpus.csv` adds BERTopic `topic_id` and UMAP coordinates
608
  (`umap2_x`, `umap2_y`, `umap3_x`, `umap3_y`, `umap3_z`) to every excerpt,
609
  making the enriched dataset independently useful for downstream analysis
610
  without re-running the pipeline.
611
 
612
+
613
 
614
  ## 12. Educational Applications
615
 
 
737
 
738
  To run the pipeline locally or contribute new analysis:
739
 
 
740
  # Clone the Space
741
  git clone https://huggingface.co/spaces/deirdosh/curriculum_analysis_german
742
  cd curriculum_analysis_german
 
746
 
747
  # Run locally
748
  python app.py
749
+
750
 
751
  A standard laptop (8 GB RAM, no GPU) can run the full pipeline in
752
  approximately 15–20 minutes on first run. GPU acceleration reduces this to