| [2026-06-11 23:35:30,563][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/config.json "HTTP/1.1 200 OK" |
| [2026-06-11 23:35:30,780][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/tokenizer_config.json "HTTP/1.1 200 OK" |
| [2026-06-11 23:35:30,989][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/models/meta-llama/Llama-2-7b-hf/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:35:31,201][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/models/meta-llama/Llama-2-7b-hf/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-11 23:35:31,270][model][INFO] - Setting pad_token as eos token: </s> |
| [2026-06-11 23:35:31,272][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/muse_Llama-2-7b-hf_News_GradDiff_lr8e-6/evals |
| [2026-06-11 23:35:31,273][evaluator][INFO] - ***** Running MUSE evaluation suite ***** |
| [2026-06-11 23:35:31,273][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/muse_Llama-2-7b-hf_News_GradDiff_lr8e-6/evals/MUSE_EVAL.json |
| [2026-06-11 23:35:31,273][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/muse_Llama-2-7b-hf_News_GradDiff_lr8e-6/evals/MUSE_SUMMARY.json |
| [2026-06-11 23:35:31,719][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:35:32,339][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:35:32,540][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:35:32,745][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/revision/506bd5b150b92814d45e4404a82f120ab2d748bf "HTTP/1.1 200 OK" |
| [2026-06-11 23:35:32,956][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:35:33,187][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:35:33,411][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf/knowmem?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-11 23:35:33,614][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf?recursive=false&expand=false "HTTP/1.1 200 OK" |
| [2026-06-11 23:35:33,821][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:35:34,079][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:35:34,286][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:35:34,506][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:35:34,720][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:35:34,941][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:35:35,154][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:35:35,160][metrics][INFO] - Evaluating forget_knowmem_ROUGE |
| [2026-06-11 23:36:00,229][evaluator][INFO] - Result for metric forget_knowmem_ROUGE: 0.4071792563550875 |
| [2026-06-11 23:36:00,456][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:36:01,022][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:36:01,236][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:36:01,453][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:36:01,694][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:36:01,903][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:36:02,111][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:36:02,301][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:36:02,614][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:36:02,826][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:36:03,043][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:36:03,252][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:36:03,258][metrics][INFO] - Evaluating retain_knowmem_ROUGE |
| [2026-06-11 23:36:13,280][evaluator][INFO] - Result for metric retain_knowmem_ROUGE: 0.3870091218723309 |
| [2026-06-11 23:36:13,496][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:36:14,070][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:36:14,273][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:36:14,488][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:36:14,728][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:36:14,936][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:36:15,151][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf/verbmem?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-11 23:36:15,156][metrics][INFO] - Evaluating forget_verbmem_ROUGE |
| [2026-06-11 23:37:56,453][evaluator][INFO] - Result for metric forget_verbmem_ROUGE: 0.023419833534971676 |
| [2026-06-11 23:37:56,683][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:37:57,296][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:37:57,499][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:37:57,713][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:37:57,951][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:37:58,166][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:37:58,380][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News/tree/506bd5b150b92814d45e4404a82f120ab2d748bf/privleak?recursive=true&expand=false "HTTP/1.1 200 OK" |
| [2026-06-11 23:37:58,611][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:37:58,815][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:37:59,019][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:37:59,234][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:37:59,473][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:37:59,687][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:37:59,695][metrics][INFO] - Loading evaluations from saves/eval/muse_News_retrain/MUSE_EVAL.json |
| [2026-06-11 23:37:59,698][metrics][INFO] - Evaluating mia_min_k |
| [2026-06-11 23:38:12,057][metrics][INFO] - Loading evaluations from saves/eval/muse_News_retrain/MUSE_EVAL.json |
| [2026-06-11 23:38:12,059][metrics][INFO] - Evaluating privleak |
| [2026-06-11 23:38:12,060][evaluator][INFO] - Result for metric privleak: 109.35070390641926 |
| [2026-06-11 23:38:12,291][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:38:12,895][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-News/muse-bench/MUSE-News.py "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:38:13,103][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:38:13,319][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/.huggingface.yaml "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:38:13,545][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-News "HTTP/1.1 200 OK" |
| [2026-06-11 23:38:13,750][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-News/resolve/506bd5b150b92814d45e4404a82f120ab2d748bf/dataset_infos.json "HTTP/1.1 404 Not Found" |
| [2026-06-11 23:38:13,755][metrics][INFO] - Evaluating extraction_strength |
| [2026-06-11 23:38:17,922][evaluator][INFO] - Result for metric extraction_strength: 0.008253968253968227 |
|
|
