jaydeepb commited on
Commit
6916d7e
Β·
verified Β·
1 Parent(s): 1210a65

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +7 -22
README.md CHANGED
@@ -1,6 +1,6 @@
1
  ---
2
  title: Predicting Memorization Before Fine-Tuning
3
- emoji: πŸ”Ž
4
  colorFrom: indigo
5
  colorTo: green
6
  sdk: gradio
@@ -12,17 +12,15 @@ license: apache-2.0
12
 
13
  # Predicting Memorization Before Fine-Tuning β€” demo
14
 
15
- Interactive companion to the paper. A classifier trained only on **pre-fine-tuning**
16
- base-model features predicts which sequences a fine-tuned model will memorize.
17
 
18
- - **Explore the held-out 1M** β€” pick a real Pythia-1.4B training sequence and see whether the
19
- fine-tuned model actually memorized it (ground truth), next to the classifier's risk score and
20
- the base-model feature values. Datasets: FineWeb, PG-19, The Stack, OpenWebMath (all public).
21
- - **Score your own text** β€” paste a passage and get the classifier's *predicted* memorization
22
  risk from base-model features. This is a forecast; there is no ground truth for arbitrary text.
23
 
24
  ## Files
25
- - `app.py` β€” Gradio app (two tabs).
26
  - `data/lookup.parquet` β€” precomputed table (prefix, suffix, memorized flag, classifier score,
27
  base-model features) sampled from the held-out 1M run (all memorized examples up to a cap plus a
28
  random sample of non-memorized).
@@ -40,24 +38,11 @@ python app.py # opens a local Gradio URL
40
  Locally the app loads Pythia-1.4B (~3 GB) at startup on a GPU if present, otherwise CPU; the
41
  `@spaces.GPU` decorator is a no-op off ZeroGPU.
42
 
43
- ## Deploy on Hugging Face Spaces (free ZeroGPU)
44
- 1. Use your **own, established HF account** (free ZeroGPU hosting requires a verified email and an
45
- account older than 30 days; up to 2 ZeroGPU Spaces). Create a new Space: **SDK = Gradio**,
46
- hardware = **ZeroGPU**.
47
- 2. Push this folder to the Space repo (the YAML header above configures it):
48
- ```bash
49
- git clone https://huggingface.co/spaces/<user>/<space> && cd <space>
50
- cp -r /path/to/demo_app/* . && git add -A && git commit -m "demo" && git push
51
- ```
52
- 3. The Space builds automatically. The Explore tab is instant. "Score your own text" runs the
53
- base-model forward+backward on a GPU that ZeroGPU attaches on demand (via the `@spaces.GPU`
54
- decorator in `app.py`), so it is fast; the free tier allows 5 GPU-minutes/day and each score
55
- takes a couple of seconds.
56
 
57
  ## Notes
58
  - The classifier uses only base-model (pre-fine-tuning) features and was trained on a separate
59
  Run-1 fine-tuning run, then evaluated here on a disjoint Run-2 run.
60
- - Qwen and Llama can be added to the Explore tab once their run-2 token sequences are regenerated.
61
 
62
  ## Data and model attribution
63
  The demo code in this repository is released under Apache-2.0. The text excerpts shown in the
 
1
  ---
2
  title: Predicting Memorization Before Fine-Tuning
3
+ authors: Jaydeep Borkar, Niloofar Mireshghallah, and David A. Smith
4
  colorFrom: indigo
5
  colorTo: green
6
  sdk: gradio
 
12
 
13
  # Predicting Memorization Before Fine-Tuning β€” demo
14
 
15
+ Interactive demo to the paper. A classifier trained only on pre-fine-tuning base-model features predicts which sequences a fine-tuned model will memorize.
 
16
 
17
+ - **Explore the held-out examples** β€” pick a random sequence from a held-out set and estimate its memorization risk.
18
+ - Datasets: FineWeb, PG-19, The Stack, OpenWebMath (all public). We do not include Enron and WildChat in the demo as they may contain sensitive text.
19
+ - **Score your own text** β€” paste a passage and get the classifier's predicted memorization
 
20
  risk from base-model features. This is a forecast; there is no ground truth for arbitrary text.
21
 
22
  ## Files
23
+ - `app.py` β€” Gradio app.
24
  - `data/lookup.parquet` β€” precomputed table (prefix, suffix, memorized flag, classifier score,
25
  base-model features) sampled from the held-out 1M run (all memorized examples up to a cap plus a
26
  random sample of non-memorized).
 
38
  Locally the app loads Pythia-1.4B (~3 GB) at startup on a GPU if present, otherwise CPU; the
39
  `@spaces.GPU` decorator is a no-op off ZeroGPU.
40
 
 
 
 
 
 
 
 
 
 
 
 
 
 
41
 
42
  ## Notes
43
  - The classifier uses only base-model (pre-fine-tuning) features and was trained on a separate
44
  Run-1 fine-tuning run, then evaluated here on a disjoint Run-2 run.
45
+
46
 
47
  ## Data and model attribution
48
  The demo code in this repository is released under Apache-2.0. The text excerpts shown in the