| [2026-06-19 21:50:34,046][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/config.json "HTTP/1.1 200 OK" |
| [2026-06-19 21:50:34,273][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/tokenizer_config.json "HTTP/1.1 200 OK" |
| [2026-06-19 21:50:34,512][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/models/meta-llama/Llama-2-7b-hf/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found" |
| [2026-06-19 21:50:34,728][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/models/meta-llama/Llama-2-7b-hf/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-19 21:50:34,796][model][INFO] - Setting pad_token as eos token: </s> |
| [2026-06-19 21:50:34,797][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/muse_Llama-2-7b-hf_News_GradDiff_lr3e-5/evals |
| [2026-06-19 21:50:34,798][evaluator][INFO] - ***** Running MUSE evaluation suite ***** |
| [2026-06-19 21:50:34,799][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/muse_Llama-2-7b-hf_News_GradDiff_lr3e-5/evals/MUSE_EVAL.json |
| [2026-06-19 21:50:34,799][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/muse_Llama-2-7b-hf_News_GradDiff_lr3e-5/evals/MUSE_SUMMARY.json |
| [2026-06-19 21:50:35,004][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/main/README.md "HTTP/1.1 307 Temporary Redirect" |
| [2026-06-19 21:50:35,008][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/datasets/muse-bench/MUSE-News/506bd5b150b92814d45e4404a82f120ab2d748bf/README.md "HTTP/1.1 200 OK" |
| [2026-06-19 21:50:35,214][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-19 21:50:35,792][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-19 21:50:36,017][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/revision/506bd5b150b92814d45e4404a82f120ab2d748bf "HTTP/1.1 200 OK" |
| [2026-06-19 21:50:36,215][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-19 21:50:36,465][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-19 21:50:36,677][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf/knowmem?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-19 21:50:36,901][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf?recursive=false&expand=false "HTTP/1.1 200 OK" |
| [2026-06-19 21:50:37,099][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-19 21:50:37,324][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/main/README.md "HTTP/1.1 307 Temporary Redirect" |
| [2026-06-19 21:50:37,329][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/datasets/muse-bench/MUSE-News/506bd5b150b92814d45e4404a82f120ab2d748bf/README.md "HTTP/1.1 200 OK" |
| [2026-06-19 21:50:37,529][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-19 21:50:37,761][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-19 21:50:37,962][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-19 21:50:38,186][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-19 21:50:38,399][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-19 21:50:38,409][metrics][INFO] - Evaluating forget_knowmem_ROUGE |
| [2026-06-20 08:50:41,810][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/config.json "HTTP/1.1 200 OK" |
| [2026-06-20 08:50:42,017][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/tokenizer_config.json "HTTP/1.1 200 OK" |
| [2026-06-20 08:50:42,225][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/models/meta-llama/Llama-2-7b-hf/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:50:42,431][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/models/meta-llama/Llama-2-7b-hf/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-20 08:50:42,497][model][INFO] - Setting pad_token as eos token: </s> |
| [2026-06-20 08:50:42,498][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/muse_Llama-2-7b-hf_News_GradDiff_lr3e-5/evals |
| [2026-06-20 08:50:42,499][evaluator][INFO] - ***** Running MUSE evaluation suite ***** |
| [2026-06-20 08:50:42,500][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/muse_Llama-2-7b-hf_News_GradDiff_lr3e-5/evals/MUSE_EVAL.json |
| [2026-06-20 08:50:42,500][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/muse_Llama-2-7b-hf_News_GradDiff_lr3e-5/evals/MUSE_SUMMARY.json |
| [2026-06-20 08:50:42,787][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:50:43,413][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:50:43,616][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:50:43,817][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/revision/506bd5b150b92814d45e4404a82f120ab2d748bf "HTTP/1.1 200 OK" |
| [2026-06-20 08:50:44,031][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:50:44,264][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:50:44,473][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf/knowmem?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-20 08:50:44,686][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf?recursive=false&expand=false "HTTP/1.1 200 OK" |
| [2026-06-20 08:50:44,892][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:50:45,115][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:50:45,322][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:50:45,524][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:50:45,735][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:50:45,953][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:50:46,161][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:50:46,173][metrics][INFO] - Evaluating forget_knowmem_ROUGE |
| [2026-06-20 08:51:10,794][evaluator][INFO] - Result for metric forget_knowmem_ROUGE: 0.35864526961723764 |
| [2026-06-20 08:51:11,012][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:51:11,622][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:51:11,829][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:51:12,042][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:51:12,345][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:51:12,554][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:51:12,776][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:51:12,978][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:51:13,231][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:51:13,437][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:51:13,657][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:51:13,860][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:51:13,872][metrics][INFO] - Evaluating retain_knowmem_ROUGE |
| [2026-06-20 08:51:23,067][evaluator][INFO] - Result for metric retain_knowmem_ROUGE: 0.2749343124539142 |
| [2026-06-20 08:51:23,480][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:51:24,079][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:51:24,311][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:51:24,522][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:51:24,738][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:51:24,946][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:51:25,154][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf/verbmem?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-20 08:51:25,167][metrics][INFO] - Evaluating forget_verbmem_ROUGE |
| [2026-06-20 08:53:03,913][evaluator][INFO] - Result for metric forget_verbmem_ROUGE: 0.024667316958041162 |
| [2026-06-20 08:53:04,168][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:53:04,765][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:53:04,972][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:53:05,184][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:53:05,420][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:53:05,627][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:53:05,833][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf/privleak?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-20 08:53:06,056][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:53:06,261][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:53:06,463][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:53:06,667][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:53:06,880][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:53:07,087][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:53:07,100][metrics][INFO] - Loading evaluations from saves/eval/muse_Llama-2-7b-hf_News_retrain/MUSE_EVAL.json |
| [2026-06-20 08:53:07,102][metrics][INFO] - Evaluating mia_min_k |
| [2026-06-20 08:53:19,400][metrics][INFO] - Loading evaluations from saves/eval/muse_Llama-2-7b-hf_News_retrain/MUSE_EVAL.json |
| [2026-06-20 08:53:19,403][metrics][INFO] - Evaluating privleak |
| [2026-06-20 08:53:19,403][evaluator][INFO] - Result for metric privleak: 89.27371954465289 |
| [2026-06-20 08:53:19,686][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:53:20,259][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:53:20,513][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:53:20,723][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:53:20,956][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-20 08:53:21,167][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-20 08:53:21,179][metrics][INFO] - Evaluating extraction_strength |
| [2026-06-20 08:53:25,364][evaluator][INFO] - Result for metric extraction_strength: 0.00920634920634918 |
|
|
