| [2026-06-10 15:44:32,799][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/config.json "HTTP/1.1 200 OK" |
| [2026-06-10 15:44:33,005][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/tokenizer_config.json "HTTP/1.1 200 OK" |
| [2026-06-10 15:44:33,213][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/models/meta-llama/Llama-2-7b-hf/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:44:33,423][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/models/meta-llama/Llama-2-7b-hf/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-10 15:44:33,490][model][INFO] - Setting pad_token as eos token: </s> |
| [2026-06-10 15:44:33,492][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/muse_Llama-2-7b-hf_News_GradDiff_lr2e-5/evals |
| [2026-06-10 15:44:33,493][evaluator][INFO] - ***** Running MUSE evaluation suite ***** |
| [2026-06-10 15:44:33,493][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/muse_Llama-2-7b-hf_News_GradDiff_lr2e-5/evals/MUSE_EVAL.json |
| [2026-06-10 15:44:33,493][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/muse_Llama-2-7b-hf_News_GradDiff_lr2e-5/evals/MUSE_SUMMARY.json |
| [2026-06-10 15:44:33,699][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:44:34,322][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:44:34,529][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:44:34,755][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/revision/506bd5b150b92814d45e4404a82f120ab2d748bf "HTTP/1.1 200 OK" |
| [2026-06-10 15:44:34,959][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:44:35,190][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:44:35,396][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf/knowmem?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-10 15:44:35,602][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf?recursive=false&expand=false "HTTP/1.1 200 OK" |
| [2026-06-10 15:44:35,810][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:44:36,030][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:44:36,236][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:44:36,444][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:44:36,652][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:44:36,870][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:44:37,074][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:44:37,080][metrics][INFO] - Evaluating forget_knowmem_ROUGE |
| [2026-06-10 15:45:02,541][evaluator][INFO] - Result for metric forget_knowmem_ROUGE: 0.24943928442398328 |
| [2026-06-10 15:45:02,760][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:45:03,319][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:45:03,524][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:45:03,729][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:45:03,972][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:45:04,181][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:45:04,388][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:45:04,573][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:45:04,820][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:45:05,037][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:45:05,259][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:45:05,467][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:45:05,473][metrics][INFO] - Evaluating retain_knowmem_ROUGE |
| [2026-06-10 15:45:14,894][evaluator][INFO] - Result for metric retain_knowmem_ROUGE: 0.12772915287386674 |
| [2026-06-10 15:45:15,118][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:45:15,681][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:45:15,921][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:45:16,143][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:45:16,373][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:45:16,580][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:45:16,785][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf/verbmem?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-10 15:45:16,790][metrics][INFO] - Evaluating forget_verbmem_ROUGE |
| [2026-06-10 15:46:56,498][evaluator][INFO] - Result for metric forget_verbmem_ROUGE: 0.006750148718031961 |
| [2026-06-10 15:46:56,761][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:46:57,320][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:46:57,529][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:46:57,738][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:46:57,967][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:46:58,200][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:46:58,406][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf/privleak?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-10 15:46:58,630][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:46:58,812][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:46:59,095][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:46:59,307][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:46:59,522][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:46:59,723][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:46:59,731][metrics][INFO] - Loading evaluations from saves/eval/muse_News_retrain/MUSE_EVAL.json |
| [2026-06-10 15:46:59,733][metrics][INFO] - Evaluating mia_min_k |
| [2026-06-10 15:47:12,026][metrics][INFO] - Loading evaluations from saves/eval/muse_News_retrain/MUSE_EVAL.json |
| [2026-06-10 15:47:12,029][metrics][INFO] - Evaluating privleak |
| [2026-06-10 15:47:12,029][evaluator][INFO] - Result for metric privleak: 82.30720737921679 |
| [2026-06-10 15:47:12,324][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:47:12,953][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:47:13,198][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:47:13,818][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:47:14,047][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-10 15:47:14,247][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-10 15:47:14,252][metrics][INFO] - Evaluating extraction_strength |
| [2026-06-10 15:47:18,414][evaluator][INFO] - Result for metric extraction_strength: 0.007936507936507908 |
|
|
