[2026-06-22 14:41:48,618][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/config.json "HTTP/1.1 200 OK"
[2026-06-22 14:41:48,830][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/tokenizer_config.json "HTTP/1.1 200 OK"
[2026-06-22 14:41:49,056][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/models/meta-llama/Llama-2-7b-hf/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
[2026-06-22 14:41:49,263][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/models/meta-llama/Llama-2-7b-hf/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
[2026-06-22 14:41:49,336][model][INFO] - Setting pad_token as eos token: </s>
[2026-06-22 14:41:49,337][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/muse_Llama-2-7b-hf_Books_GradDiff_lr5e-5_alpha5/evals
[2026-06-22 14:41:49,338][evaluator][INFO] - ***** Running MUSE evaluation suite *****
[2026-06-22 14:41:49,338][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/muse_Llama-2-7b-hf_Books_GradDiff_lr5e-5_alpha5/evals/MUSE_EVAL.json
[2026-06-22 14:41:49,338][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/muse_Llama-2-7b-hf_Books_GradDiff_lr5e-5_alpha5/evals/MUSE_SUMMARY.json
[2026-06-22 14:41:49,542][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:41:50,161][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found"
[2026-06-22 14:41:50,370][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:41:50,593][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books/revision/051ba90319e920d410d87cfdbd61f25843c1b892 "HTTP/1.1 200 OK"
[2026-06-22 14:41:50,799][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found"
[2026-06-22 14:41:51,035][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:41:51,241][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books/tree/051ba90319e920d410d87cfdbd61f25843c1b892/knowmem?recursive=true&expand=false "HTTP/1.1 200 OK"
[2026-06-22 14:41:51,442][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books/tree/051ba90319e920d410d87cfdbd61f25843c1b892?recursive=false&expand=false "HTTP/1.1 200 OK"
[2026-06-22 14:41:51,647][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found"
[2026-06-22 14:41:51,871][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/knowmem/retain_qa_icl-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:41:52,093][httpx][INFO] - HTTP Request: GET https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/knowmem/retain_qa_icl-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:41:52,299][httpx][INFO] - HTTP Request: GET https://cas-bridge.xethub.hf.co/xet-bridge-us/666a6220e61e886fb70270a7/682779f2978f1d5b9a1145d82cdffe69bf5ad5cb0d79436fb0a9b8c511223d00?Expires=1782110511&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYXMtYnJpZGdlLnhldGh1Yi5oZi5jby94ZXQtYnJpZGdlLXVzLzY2NmE2MjIwZTYxZTg4NmZiNzAyNzBhNy82ODI3NzlmMjk3OGYxZDViOWExMTQ1ZDgyY2RmZmU2OWJmNWFkNWNiMGQ3OTQzNmZiMGE5YjhjNTExMjIzZDAwKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc4MjExMDUxMX19fV19&Signature=MEUCIHliFZrqM9zl7rokG6byT71SUp9eed-2-9CxjTbSCvp5AiEAt3O1KVzUJjTYVdr68bCnnr8X4YzxUMrLwwT2e5Jw71I_&Key-Pair-Id=K1LYXO563TGWFU&response-content-disposition=inline%3B+filename*%3DUTF-8%27%27retain_qa_icl-00000-of-00001.parquet%3B+filename%3D%22retain_qa_icl-00000-of-00001.parquet%22%3B&X-Xet-Cas-Uid=66b0c434a210bfb3f8bba264&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=cas%2F20260622%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260622T054151Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=8f24665dbf656181613b2b61f0f2132cdad0ebf584b955edcb2747f390a839f4 "HTTP/1.1 200 OK"
[2026-06-22 14:41:52,508][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/knowmem/retain_qa-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:41:52,720][httpx][INFO] - HTTP Request: GET https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/knowmem/retain_qa-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:41:52,912][httpx][INFO] - HTTP Request: GET https://cas-bridge.xethub.hf.co/xet-bridge-us/666a6220e61e886fb70270a7/45aa03626d9f35528c238b9e52428a32d6553c891967895b4faf6315f401b2ea?Expires=1782110512&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYXMtYnJpZGdlLnhldGh1Yi5oZi5jby94ZXQtYnJpZGdlLXVzLzY2NmE2MjIwZTYxZTg4NmZiNzAyNzBhNy80NWFhMDM2MjZkOWYzNTUyOGMyMzhiOWU1MjQyOGEzMmQ2NTUzYzg5MTk2Nzg5NWI0ZmFmNjMxNWY0MDFiMmVhKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc4MjExMDUxMn19fV19&Signature=MEQCIFG8Q5d%7EktWN0btWJi7CZS6zrhihVk2sWQfKQ3aVVvqkAiBau1OJsJW9zT%7EJ1NS8vM3lIDa0N9Xtf03soav-kBMA3w__&Key-Pair-Id=K1LYXO563TGWFU&response-content-disposition=inline%3B+filename*%3DUTF-8%27%27retain_qa-00000-of-00001.parquet%3B+filename%3D%22retain_qa-00000-of-00001.parquet%22%3B&X-Xet-Cas-Uid=66b0c434a210bfb3f8bba264&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=cas%2F20260622%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260622T054152Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=0e823036fc863ecb263634911e5eb99b93c8efd0e8262b4edd2f62c116be37fa "HTTP/1.1 200 OK"
[2026-06-22 14:41:53,127][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/knowmem/forget_qa-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:41:53,349][httpx][INFO] - HTTP Request: GET https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/knowmem/forget_qa-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:41:53,542][httpx][INFO] - HTTP Request: GET https://cas-bridge.xethub.hf.co/xet-bridge-us/666a6220e61e886fb70270a7/02511061301b8ee4f50d533c2b457f5cc39c666afe09c6c9f33c724a203a23f0?Expires=1782110513&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYXMtYnJpZGdlLnhldGh1Yi5oZi5jby94ZXQtYnJpZGdlLXVzLzY2NmE2MjIwZTYxZTg4NmZiNzAyNzBhNy8wMjUxMTA2MTMwMWI4ZWU0ZjUwZDUzM2MyYjQ1N2Y1Y2MzOWM2NjZhZmUwOWM2YzlmMzNjNzI0YTIwM2EyM2YwKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc4MjExMDUxM319fV19&Signature=MEUCIAkJC-UEHjZ1QF7bBylKPDxZ-R1HJecz70f4NylSYO3fAiEA6GRg7PtCKDJoXr1Vfsr6axHuZc088-mRHMU9WUMPuEU_&Key-Pair-Id=K1LYXO563TGWFU&X-Xet-Cas-Uid=66b0c434a210bfb3f8bba264&response-content-disposition=inline%3B+filename*%3DUTF-8%27%27forget_qa-00000-of-00001.parquet%3B+filename%3D%22forget_qa-00000-of-00001.parquet%22%3B&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=cas%2F20260622%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260622T054153Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=99210edc8eec57b1036171308b482b4714b4cce80107e26ec9a1dee2b8a0a194 "HTTP/1.1 200 OK"
[2026-06-22 14:41:53,758][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/knowmem/forget_qa_icl-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:41:53,988][httpx][INFO] - HTTP Request: GET https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/knowmem/forget_qa_icl-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:41:54,184][httpx][INFO] - HTTP Request: GET https://cas-bridge.xethub.hf.co/xet-bridge-us/666a6220e61e886fb70270a7/e40499f157f102da0e4d4a335ec744ecd75810765debd3800d616a95fe8b1a60?Expires=1782110513&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYXMtYnJpZGdlLnhldGh1Yi5oZi5jby94ZXQtYnJpZGdlLXVzLzY2NmE2MjIwZTYxZTg4NmZiNzAyNzBhNy9lNDA0OTlmMTU3ZjEwMmRhMGU0ZDRhMzM1ZWM3NDRlY2Q3NTgxMDc2NWRlYmQzODAwZDYxNmE5NWZlOGIxYTYwKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc4MjExMDUxM319fV19&Signature=MEYCIQC00TuZr70zx7hxpBpVl3iOrG5v-e3y1BNJeSmcJ2yTsgIhAOyzmOk6p9a3MQNpSFeOp-bMpy9iCjeVeeIqU8iIlppt&Key-Pair-Id=K1LYXO563TGWFU&response-content-disposition=inline%3B+filename*%3DUTF-8%27%27forget_qa_icl-00000-of-00001.parquet%3B+filename%3D%22forget_qa_icl-00000-of-00001.parquet%22%3B&X-Xet-Cas-Uid=66b0c434a210bfb3f8bba264&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=cas%2F20260622%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260622T054153Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d4d0c07ad26f3b77d088c365f6f88d287d449635f00d17682055b6b2b3ec7bbf "HTTP/1.1 200 OK"
[2026-06-22 14:41:54,419][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:41:54,624][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found"
[2026-06-22 14:41:54,828][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:41:55,040][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found"
[2026-06-22 14:41:55,259][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:41:55,465][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found"
[2026-06-22 14:41:55,479][metrics][INFO] - Evaluating forget_knowmem_ROUGE
[2026-06-22 14:42:23,869][evaluator][INFO] - Result for metric forget_knowmem_ROUGE:	0.0
[2026-06-22 14:42:24,094][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:42:24,701][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found"
[2026-06-22 14:42:24,908][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:42:25,112][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found"
[2026-06-22 14:42:25,349][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:42:25,557][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found"
[2026-06-22 14:42:25,832][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:42:26,037][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found"
[2026-06-22 14:42:26,239][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:42:26,454][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found"
[2026-06-22 14:42:26,673][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:42:26,877][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found"
[2026-06-22 14:42:26,890][metrics][INFO] - Evaluating retain_knowmem_ROUGE
[2026-06-22 14:42:51,068][evaluator][INFO] - Result for metric retain_knowmem_ROUGE:	0.0
[2026-06-22 14:42:51,287][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:42:51,847][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found"
[2026-06-22 14:42:52,058][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:42:52,267][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found"
[2026-06-22 14:42:52,498][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:42:52,707][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found"
[2026-06-22 14:42:52,924][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books/tree/051ba90319e920d410d87cfdbd61f25843c1b892/verbmem?recursive=true&expand=false "HTTP/1.1 200 OK"
[2026-06-22 14:42:53,153][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/verbmem/forget-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:42:53,371][httpx][INFO] - HTTP Request: GET https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/verbmem/forget-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:42:53,597][httpx][INFO] - HTTP Request: GET https://cas-bridge.xethub.hf.co/xet-bridge-us/666a6220e61e886fb70270a7/a0e406a83cec52b368b732f61054fba0a154644c7dd9b32f354c718607812c3f?Expires=1782110573&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYXMtYnJpZGdlLnhldGh1Yi5oZi5jby94ZXQtYnJpZGdlLXVzLzY2NmE2MjIwZTYxZTg4NmZiNzAyNzBhNy9hMGU0MDZhODNjZWM1MmIzNjhiNzMyZjYxMDU0ZmJhMGExNTQ2NDRjN2RkOWIzMmYzNTRjNzE4NjA3ODEyYzNmKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc4MjExMDU3M319fV19&Signature=MEUCIQD-%7EfON5pfKlAA8pHklF01gpvdzrC4PGmVC1IVjl7PRlAIgCpv70dlgrlnnt3GcZfqZSG0yXgCKfpxVG0x0yuOLe8Q_&Key-Pair-Id=K1LYXO563TGWFU&response-content-disposition=inline%3B+filename*%3DUTF-8%27%27forget-00000-of-00001.parquet%3B+filename%3D%22forget-00000-of-00001.parquet%22%3B&X-Xet-Cas-Uid=66b0c434a210bfb3f8bba264&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=cas%2F20260622%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260622T054253Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=ad7f414a8e9c6483cc03cf6f3192fb4b3deb98bb31a8a3e7829143f46ffd1853 "HTTP/1.1 200 OK"
[2026-06-22 14:42:53,682][metrics][INFO] - Evaluating forget_verbmem_ROUGE
[2026-06-22 14:44:42,843][evaluator][INFO] - Result for metric forget_verbmem_ROUGE:	0.0
[2026-06-22 14:44:43,070][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:44:43,692][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found"
[2026-06-22 14:44:43,897][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:44:44,117][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found"
[2026-06-22 14:44:44,489][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:44:44,695][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found"
[2026-06-22 14:44:44,953][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books/tree/051ba90319e920d410d87cfdbd61f25843c1b892/privleak?recursive=true&expand=false "HTTP/1.1 200 OK"
[2026-06-22 14:44:45,174][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/privleak/retain-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:44:45,389][httpx][INFO] - HTTP Request: GET https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/privleak/retain-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:44:45,594][httpx][INFO] - HTTP Request: GET https://cas-bridge.xethub.hf.co/xet-bridge-us/666a6220e61e886fb70270a7/805eb496ee9a809c63111c9b69bb61442547638657024672e4e3edbc613058ab?Expires=1782110685&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYXMtYnJpZGdlLnhldGh1Yi5oZi5jby94ZXQtYnJpZGdlLXVzLzY2NmE2MjIwZTYxZTg4NmZiNzAyNzBhNy84MDVlYjQ5NmVlOWE4MDljNjMxMTFjOWI2OWJiNjE0NDI1NDc2Mzg2NTcwMjQ2NzJlNGUzZWRiYzYxMzA1OGFiKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc4MjExMDY4NX19fV19&Signature=MEQCIGKY0VEjLRib0OCLexwcR1jZ5ExgSw2u7Kciw%7El-473%7EAiAjy-aSUhb76pfKCtoE4rTu6B3SW0CXNvWYoaa7t5uCsQ__&Key-Pair-Id=K1LYXO563TGWFU&X-Xet-Cas-Uid=66b0c434a210bfb3f8bba264&response-content-disposition=inline%3B+filename*%3DUTF-8%27%27retain-00000-of-00001.parquet%3B+filename%3D%22retain-00000-of-00001.parquet%22%3B&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=cas%2F20260622%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260622T054445Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d0e8ee855dc308331fe9181af0b06ea141723f425342a09cf0eedfd98cd2763d "HTTP/1.1 200 OK"
[2026-06-22 14:44:45,833][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/privleak/forget-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:44:46,072][httpx][INFO] - HTTP Request: GET https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/privleak/forget-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:44:46,269][httpx][INFO] - HTTP Request: GET https://cas-bridge.xethub.hf.co/xet-bridge-us/666a6220e61e886fb70270a7/601c7685f98ed3b2cac19ec34e2419a89328b4cd421ed7e82d0749e16495c85d?Expires=1782110685&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYXMtYnJpZGdlLnhldGh1Yi5oZi5jby94ZXQtYnJpZGdlLXVzLzY2NmE2MjIwZTYxZTg4NmZiNzAyNzBhNy82MDFjNzY4NWY5OGVkM2IyY2FjMTllYzM0ZTI0MTlhODkzMjhiNGNkNDIxZWQ3ZTgyZDA3NDllMTY0OTVjODVkKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc4MjExMDY4NX19fV19&Signature=MEUCIFEVIY0vZLkrFjkbb6uYKEsbCQS9jXlJnF1l25gzwnkBAiEAhr4sQtUd9yZETPELWWoZkQ6z3QqcCZLJNPiBq6tegt8_&Key-Pair-Id=K1LYXO563TGWFU&X-Xet-Cas-Uid=66b0c434a210bfb3f8bba264&response-content-disposition=inline%3B+filename*%3DUTF-8%27%27forget-00000-of-00001.parquet%3B+filename%3D%22forget-00000-of-00001.parquet%22%3B&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=cas%2F20260622%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260622T054445Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=9072c377d3fdc861e7e8437393c3c1a0901f7e9ec7439f10b5bfd32bd5561d50 "HTTP/1.1 200 OK"
[2026-06-22 14:44:46,502][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/privleak/holdout-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:44:46,716][httpx][INFO] - HTTP Request: GET https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/privleak/holdout-00000-of-00001.parquet "HTTP/1.1 302 Found"
[2026-06-22 14:44:46,904][httpx][INFO] - HTTP Request: GET https://cas-bridge.xethub.hf.co/xet-bridge-us/666a6220e61e886fb70270a7/25f2737b8b9985074e02765ff915411c258858c5a5ce7a38ee4cc05c61d196bd?Expires=1782110686&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYXMtYnJpZGdlLnhldGh1Yi5oZi5jby94ZXQtYnJpZGdlLXVzLzY2NmE2MjIwZTYxZTg4NmZiNzAyNzBhNy8yNWYyNzM3YjhiOTk4NTA3NGUwMjc2NWZmOTE1NDExYzI1ODg1OGM1YTVjZTdhMzhlZTRjYzA1YzYxZDE5NmJkKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc4MjExMDY4Nn19fV19&Signature=MEYCIQD60T1ZfCVIrVlOQRtKrw5ooP7Q6ybIZZGtpYeHHm7edgIhANQ29HjaCkbJFSAFSe0l5AG8%7ELy-K3I4idGw%7EMDWymku&Key-Pair-Id=K1LYXO563TGWFU&X-Xet-Cas-Uid=66b0c434a210bfb3f8bba264&response-content-disposition=inline%3B+filename*%3DUTF-8%27%27holdout-00000-of-00001.parquet%3B+filename%3D%22holdout-00000-of-00001.parquet%22%3B&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=cas%2F20260622%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260622T054446Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=84c3ced9b5640808656d183492e6eb4fc8bb45a955bbb562b7dde82e0347c8d7 "HTTP/1.1 200 OK"
[2026-06-22 14:44:47,167][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:44:47,373][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found"
[2026-06-22 14:44:47,626][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:44:47,847][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found"
[2026-06-22 14:44:48,059][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:44:48,264][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found"
[2026-06-22 14:44:48,278][metrics][INFO] - Loading evaluations from saves/eval/muse_Llama-2-7b-hf_Books_retrain/MUSE_EVAL.json
[2026-06-22 14:44:48,282][metrics][INFO] - Evaluating mia_min_k
[2026-06-22 14:45:02,676][metrics][INFO] - Loading evaluations from saves/eval/muse_Llama-2-7b-hf_Books_retrain/MUSE_EVAL.json
[2026-06-22 14:45:02,679][metrics][INFO] - Evaluating privleak
[2026-06-22 14:45:02,679][evaluator][INFO] - Result for metric privleak:	-10.179363903443193
[2026-06-22 14:45:02,985][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:45:03,583][httpx][INFO] - HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/muse-bench/MUSE-Books/muse-bench/MUSE-Books.py "HTTP/1.1 404 Not Found"
[2026-06-22 14:45:03,794][httpx][INFO] - HTTP Request: GET https://huggingface.co/api/datasets/muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:45:04,004][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/.huggingface.yaml "HTTP/1.1 404 Not Found"
[2026-06-22 14:45:04,239][httpx][INFO] - HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=muse-bench/MUSE-Books "HTTP/1.1 200 OK"
[2026-06-22 14:45:04,443][httpx][INFO] - HTTP Request: HEAD https://huggingface.co/datasets/muse-bench/MUSE-Books/resolve/051ba90319e920d410d87cfdbd61f25843c1b892/dataset_infos.json "HTTP/1.1 404 Not Found"
[2026-06-22 14:45:04,468][metrics][INFO] - Evaluating extraction_strength
[2026-06-22 14:45:14,072][evaluator][INFO] - Result for metric extraction_strength:	0.007936507936507908