Spaces:
Sleeping
Sleeping
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
---
|
| 2 |
title: Predicting Memorization Before Fine-Tuning
|
| 3 |
-
|
| 4 |
colorFrom: indigo
|
| 5 |
colorTo: green
|
| 6 |
sdk: gradio
|
|
@@ -12,17 +12,15 @@ license: apache-2.0
|
|
| 12 |
|
| 13 |
# Predicting Memorization Before Fine-Tuning β demo
|
| 14 |
|
| 15 |
-
Interactive
|
| 16 |
-
base-model features predicts which sequences a fine-tuned model will memorize.
|
| 17 |
|
| 18 |
-
- **Explore the held-out
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
- **Score your own text** β paste a passage and get the classifier's *predicted* memorization
|
| 22 |
risk from base-model features. This is a forecast; there is no ground truth for arbitrary text.
|
| 23 |
|
| 24 |
## Files
|
| 25 |
-
- `app.py` β Gradio app
|
| 26 |
- `data/lookup.parquet` β precomputed table (prefix, suffix, memorized flag, classifier score,
|
| 27 |
base-model features) sampled from the held-out 1M run (all memorized examples up to a cap plus a
|
| 28 |
random sample of non-memorized).
|
|
@@ -40,24 +38,11 @@ python app.py # opens a local Gradio URL
|
|
| 40 |
Locally the app loads Pythia-1.4B (~3 GB) at startup on a GPU if present, otherwise CPU; the
|
| 41 |
`@spaces.GPU` decorator is a no-op off ZeroGPU.
|
| 42 |
|
| 43 |
-
## Deploy on Hugging Face Spaces (free ZeroGPU)
|
| 44 |
-
1. Use your **own, established HF account** (free ZeroGPU hosting requires a verified email and an
|
| 45 |
-
account older than 30 days; up to 2 ZeroGPU Spaces). Create a new Space: **SDK = Gradio**,
|
| 46 |
-
hardware = **ZeroGPU**.
|
| 47 |
-
2. Push this folder to the Space repo (the YAML header above configures it):
|
| 48 |
-
```bash
|
| 49 |
-
git clone https://huggingface.co/spaces/<user>/<space> && cd <space>
|
| 50 |
-
cp -r /path/to/demo_app/* . && git add -A && git commit -m "demo" && git push
|
| 51 |
-
```
|
| 52 |
-
3. The Space builds automatically. The Explore tab is instant. "Score your own text" runs the
|
| 53 |
-
base-model forward+backward on a GPU that ZeroGPU attaches on demand (via the `@spaces.GPU`
|
| 54 |
-
decorator in `app.py`), so it is fast; the free tier allows 5 GPU-minutes/day and each score
|
| 55 |
-
takes a couple of seconds.
|
| 56 |
|
| 57 |
## Notes
|
| 58 |
- The classifier uses only base-model (pre-fine-tuning) features and was trained on a separate
|
| 59 |
Run-1 fine-tuning run, then evaluated here on a disjoint Run-2 run.
|
| 60 |
-
|
| 61 |
|
| 62 |
## Data and model attribution
|
| 63 |
The demo code in this repository is released under Apache-2.0. The text excerpts shown in the
|
|
|
|
| 1 |
---
|
| 2 |
title: Predicting Memorization Before Fine-Tuning
|
| 3 |
+
authors: Jaydeep Borkar, Niloofar Mireshghallah, and David A. Smith
|
| 4 |
colorFrom: indigo
|
| 5 |
colorTo: green
|
| 6 |
sdk: gradio
|
|
|
|
| 12 |
|
| 13 |
# Predicting Memorization Before Fine-Tuning β demo
|
| 14 |
|
| 15 |
+
Interactive demo to the paper. A classifier trained only on pre-fine-tuning base-model features predicts which sequences a fine-tuned model will memorize.
|
|
|
|
| 16 |
|
| 17 |
+
- **Explore the held-out examples** β pick a random sequence from a held-out set and estimate its memorization risk.
|
| 18 |
+
- Datasets: FineWeb, PG-19, The Stack, OpenWebMath (all public). We do not include Enron and WildChat in the demo as they may contain sensitive text.
|
| 19 |
+
- **Score your own text** β paste a passage and get the classifier's predicted memorization
|
|
|
|
| 20 |
risk from base-model features. This is a forecast; there is no ground truth for arbitrary text.
|
| 21 |
|
| 22 |
## Files
|
| 23 |
+
- `app.py` β Gradio app.
|
| 24 |
- `data/lookup.parquet` β precomputed table (prefix, suffix, memorized flag, classifier score,
|
| 25 |
base-model features) sampled from the held-out 1M run (all memorized examples up to a cap plus a
|
| 26 |
random sample of non-memorized).
|
|
|
|
| 38 |
Locally the app loads Pythia-1.4B (~3 GB) at startup on a GPU if present, otherwise CPU; the
|
| 39 |
`@spaces.GPU` decorator is a no-op off ZeroGPU.
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
|
| 42 |
## Notes
|
| 43 |
- The classifier uses only base-model (pre-fine-tuning) features and was trained on a separate
|
| 44 |
Run-1 fine-tuning run, then evaluated here on a disjoint Run-2 run.
|
| 45 |
+
|
| 46 |
|
| 47 |
## Data and model attribution
|
| 48 |
The demo code in this repository is released under Apache-2.0. The text excerpts shown in the
|