| [2026-06-12 18:28:04,009][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/config.json "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:04,219][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/tokenizer_config.json "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:04,428][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/models/meta-llama/Llama-2-7b-hf/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:28:04,639][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/models/meta-llama/Llama-2-7b-hf/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:04,705][model][INFO] - Setting pad_token as eos token: </s> |
| [2026-06-12 18:28:04,707][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/muse_Llama-2-7b-hf_Books_GradDiff_lr1e-5/evals |
| [2026-06-12 18:28:04,708][evaluator][INFO] - ***** Running MUSE evaluation suite ***** |
| [2026-06-12 18:28:04,708][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/muse_Llama-2-7b-hf_Books_GradDiff_lr1e-5/evals/MUSE_EVAL.json |
| [2026-06-12 18:28:04,708][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/muse_Llama-2-7b-hf_Books_GradDiff_lr1e-5/evals/MUSE_SUMMARY.json |
| [2026-06-12 18:28:04,918][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:05,551][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:28:05,761][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:05,969][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books/revision/051ba90319e920d410d87cfdbd61f25843c1b892 "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:06,178][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:28:06,397][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:06,615][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books/tree/051ba90319e920d410d87cfdbd61f25843c1b892/knowmem?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:06,824][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books/tree/051ba90319e920d410d87cfdbd61f25843c1b892?recursive=false&expand=false "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:07,039][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:28:07,262][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:07,473][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:28:07,673][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:07,889][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:28:08,104][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:08,310][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:28:08,318][metrics][INFO] - Evaluating forget_knowmem_ROUGE |
| [2026-06-12 18:28:36,702][evaluator][INFO] - Result for metric forget_knowmem_ROUGE: 0.0 |
| [2026-06-12 18:28:36,923][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:37,508][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:28:37,761][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:38,071][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:28:38,301][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:38,508][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:28:39,118][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:39,318][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:28:39,530][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:39,735][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:28:39,959][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:28:40,163][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:28:40,167][metrics][INFO] - Evaluating retain_knowmem_ROUGE |
| [2026-06-12 18:29:04,203][evaluator][INFO] - Result for metric retain_knowmem_ROUGE: 0.0 |
| [2026-06-12 18:29:04,553][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:29:05,117][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:29:05,318][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:29:05,529][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:29:05,805][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:29:06,008][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:29:06,231][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books/tree/051ba90319e920d410d87cfdbd61f25843c1b892/verbmem?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-12 18:29:06,238][metrics][INFO] - Evaluating forget_verbmem_ROUGE |
| [2026-06-12 18:30:55,497][evaluator][INFO] - Result for metric forget_verbmem_ROUGE: 0.0 |
| [2026-06-12 18:30:55,717][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:30:56,356][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:30:56,570][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:30:56,782][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:30:57,042][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:30:57,635][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:30:57,884][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books/tree/051ba90319e920d410d87cfdbd61f25843c1b892/privleak?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-12 18:30:58,096][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:30:58,305][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:30:58,512][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:30:58,730][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:30:58,938][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:30:59,152][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:30:59,161][metrics][INFO] - Loading evaluations from saves/eval/muse_Llama-2-7b-hf_Books_retrain/MUSE_EVAL.json |
| [2026-06-12 18:30:59,164][metrics][INFO] - Evaluating mia_min_k |
| [2026-06-12 18:31:13,381][metrics][INFO] - Loading evaluations from saves/eval/muse_Llama-2-7b-hf_Books_retrain/MUSE_EVAL.json |
| [2026-06-12 18:31:13,387][metrics][INFO] - Evaluating privleak |
| [2026-06-12 18:31:13,387][evaluator][INFO] - Result for metric privleak: -24.001479285502707 |
| [2026-06-12 18:31:13,608][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:31:14,185][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:31:14,394][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:31:14,594][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:31:14,856][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK" |
| [2026-06-12 18:31:15,077][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-12 18:31:15,082][metrics][INFO] - Evaluating extraction_strength |
| [2026-06-12 18:31:24,896][evaluator][INFO] - Result for metric extraction_strength: 0.007936507936507908 |
|
|
