Instructions to use Gege24/environment_test_affine with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries
PEFT
How to use Gege24/environment_test_affine with PEFT:
```
Base model is not found.
```

How to use Gege24/environment_test_affine with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Gege24/environment_test_affine")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Gege24/environment_test_affine")
model = AutoModelForCausalLM.from_pretrained("Gege24/environment_test_affine")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Gege24/environment_test_affine with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Gege24/environment_test_affine"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Gege24/environment_test_affine",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Gege24/environment_test_affine

SGLang

How to use Gege24/environment_test_affine with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Gege24/environment_test_affine" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Gege24/environment_test_affine",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Gege24/environment_test_affine" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Gege24/environment_test_affine",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Gege24/environment_test_affine with Docker Model Runner:
```
docker model run hf.co/Gege24/environment_test_affine
```

Gege24 commited on Feb 8

Commit

1802ce3

verified ·

1 Parent(s): a807cb0

Upload task output 1

Browse files

Files changed (4) hide show

loss.txt +1 -1
model.safetensors +1 -1
trainer_state.json +370 -370
training_args.bin +1 -1

loss.txt CHANGED Viewed

	@@ -1 +1 @@
1	- 75,-0.~~06899999883025884~~


1	+ 75,-0.12000000476837158

model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d37254765789231f8a3a2350286dac9a3200c0359f893e954337a3ad0438489f
 size 988097824

 version https://git-lfs.github.com/spec/v1
+oid sha256:d495dcb537ad13a85d3c74235ea2ad8f46917cd31620e9644884473efb2fcc9d
 size 988097824

trainer_state.json CHANGED Viewed

@@ -2,7 +2,7 @@
   "best_global_step": null,
   "best_metric": null,
   "best_model_checkpoint": null,
-  "epoch": 0.01875,
   "eval_steps": 500,
   "global_step": 75,
   "is_hyper_param_search": false,
@@ -16,31 +16,31 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
-      "completions/max_length": 374.4,
-      "completions/max_terminated_length": 374.4,
-      "completions/mean_length": 290.6960021972656,
-      "completions/mean_terminated_length": 290.6960021972656,
-      "completions/min_length": 167.6,
-      "completions/min_terminated_length": 167.6,
-      "entropy": 0.7587062597274781,
-      "epoch": 0.00125,
-      "frac_reward_zero_std": 0.6159999966621399,
-      "grad_norm": 0.32421875,
-      "kl": 0.007210364751517773,
       "learning_rate": 1.137216e-06,
-      "loss": -0.000457207765430212,
-      "num_tokens": 211069.0,
-      "reward": 0.06548000425100327,
-      "reward_std": 0.0959968164563179,
-      "rewards/env_goofspiel_reward/mean": 0.06548000574111938,
-      "rewards/env_goofspiel_reward/std": 0.20242175161838533,
-      "sampling/importance_sampling_ratio/max": 1.9353477716445924,
-      "sampling/importance_sampling_ratio/mean": 1.0085526347160338,
-      "sampling/importance_sampling_ratio/min": 0.5233797252178192,
-      "sampling/sampling_logp_difference/max": 0.6646340131759644,
-      "sampling/sampling_logp_difference/mean": 0.06287899985909462,
       "step": 5,
-      "step_time": 5.862163845000214
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -51,29 +51,29 @@
       "completions/clipped_ratio": 0.0,
       "completions/max_length": 373.8,
       "completions/max_terminated_length": 373.8,
-      "completions/mean_length": 285.77599182128904,
-      "completions/mean_terminated_length": 285.77599182128904,
-      "completions/min_length": 187.0,
-      "completions/min_terminated_length": 187.0,
-      "entropy": 0.7223453521728516,
-      "epoch": 0.0025,
-      "frac_reward_zero_std": 0.6879999876022339,
-      "grad_norm": 0.373046875,
-      "kl": 0.01190796154551208,
       "learning_rate": 2.5587359999999995e-06,
-      "loss": 0.0004954038187861443,
-      "num_tokens": 419291.0,
-      "reward": 0.06636000201106071,
-      "reward_std": 0.08932172805070877,
-      "rewards/env_goofspiel_reward/mean": 0.06636000350117684,
-      "rewards/env_goofspiel_reward/std": 0.20822030901908875,
-      "sampling/importance_sampling_ratio/max": 1.9151880025863648,
-      "sampling/importance_sampling_ratio/mean": 1.00549418926239,
-      "sampling/importance_sampling_ratio/min": 0.5084045708179474,
-      "sampling/sampling_logp_difference/max": 0.8484642028808593,
-      "sampling/sampling_logp_difference/mean": 0.05378681421279907,
       "step": 10,
-      "step_time": 5.392790380800034
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -82,31 +82,31 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
-      "completions/max_length": 374.0,
-      "completions/max_terminated_length": 374.0,
-      "completions/mean_length": 286.59998779296876,
-      "completions/mean_terminated_length": 286.59998779296876,
-      "completions/min_length": 186.4,
-      "completions/min_terminated_length": 186.4,
-      "entropy": 0.7013472616672516,
-      "epoch": 0.00375,
-      "frac_reward_zero_std": 0.743999981880188,
-      "grad_norm": 0.25,
-      "kl": 0.03683878714218736,
       "learning_rate": 3.9802559999999995e-06,
-      "loss": 0.00018892403459176422,
-      "num_tokens": 629381.0,
-      "reward": 0.05708000063896179,
-      "reward_std": 0.0819678172469139,
-      "rewards/env_goofspiel_reward/mean": 0.05708000287413597,
-      "rewards/env_goofspiel_reward/std": 0.19145022630691527,
-      "sampling/importance_sampling_ratio/max": 1.9665297031402589,
-      "sampling/importance_sampling_ratio/mean": 1.0253857135772706,
-      "sampling/importance_sampling_ratio/min": 0.538357138633728,
-      "sampling/sampling_logp_difference/max": 0.7756662368774414,
-      "sampling/sampling_logp_difference/mean": 0.06448477879166603,
       "step": 15,
-      "step_time": 5.349944851799956
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -115,31 +115,31 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
-      "completions/max_length": 374.8,
-      "completions/max_terminated_length": 374.8,
-      "completions/mean_length": 291.7519897460937,
-      "completions/mean_terminated_length": 291.7519897460937,
       "completions/min_length": 199.6,
       "completions/min_terminated_length": 199.6,
-      "entropy": 0.7047547340393067,
-      "epoch": 0.005,
-      "frac_reward_zero_std": 0.7519999742507935,
-      "grad_norm": 0.294921875,
-      "kl": 0.08839112929999829,
       "learning_rate": 5.401775999999999e-06,
-      "loss": -2.657829609233886e-05,
-      "num_tokens": 840556.0,
-      "reward": 0.06196000054478645,
-      "reward_std": 0.07529272958636284,
-      "rewards/env_goofspiel_reward/mean": 0.061960003525018695,
-      "rewards/env_goofspiel_reward/std": 0.19518478214740753,
-      "sampling/importance_sampling_ratio/max": 1.6097611427307128,
-      "sampling/importance_sampling_ratio/mean": 0.9887367606163024,
-      "sampling/importance_sampling_ratio/min": 0.46161189675331116,
-      "sampling/sampling_logp_difference/max": 0.6100110828876495,
-      "sampling/sampling_logp_difference/mean": 0.058672596514225,
       "step": 20,
-      "step_time": 5.242961023999942
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -148,31 +148,31 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
-      "completions/max_length": 374.2,
-      "completions/max_terminated_length": 374.2,
-      "completions/mean_length": 287.9799865722656,
-      "completions/mean_terminated_length": 287.9799865722656,
-      "completions/min_length": 194.4,
-      "completions/min_terminated_length": 194.4,
-      "entropy": 0.6883363842964172,
-      "epoch": 0.00625,
-      "frac_reward_zero_std": 0.7599999785423279,
-      "grad_norm": 0.1826171875,
-      "kl": 0.13179944828152657,
       "learning_rate": 6.8232959999999994e-06,
-      "loss": 6.652346346527337e-05,
-      "num_tokens": 1049402.0,
-      "reward": 0.07416000291705131,
-      "reward_std": 0.10555689632892609,
-      "rewards/env_goofspiel_reward/mean": 0.07416000291705131,
-      "rewards/env_goofspiel_reward/std": 0.22879809141159058,
-      "sampling/importance_sampling_ratio/max": 1.9118820905685425,
-      "sampling/importance_sampling_ratio/mean": 0.9868301272392273,
-      "sampling/importance_sampling_ratio/min": 0.6072056829929352,
-      "sampling/sampling_logp_difference/max": 0.5817384719848633,
-      "sampling/sampling_logp_difference/mean": 0.05620769709348679,
       "step": 25,
-      "step_time": 5.260266781999962
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -181,31 +181,31 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
-      "completions/max_length": 416.8,
-      "completions/max_terminated_length": 416.8,
-      "completions/mean_length": 299.07598876953125,
-      "completions/mean_terminated_length": 299.07598876953125,
       "completions/min_length": 212.0,
       "completions/min_terminated_length": 212.0,
-      "entropy": 0.4900937914848328,
-      "epoch": 0.0075,
-      "frac_reward_zero_std": 0.7599999785423279,
-      "grad_norm": 0.1318359375,
-      "kl": 0.13062527999281884,
       "learning_rate": 8.244816e-06,
-      "loss": 0.00041568251326680184,
-      "num_tokens": 1263196.0,
-      "reward": 0.07428000047802925,
-      "reward_std": 0.10527405217289924,
-      "rewards/env_goofspiel_reward/mean": 0.07428000047802925,
-      "rewards/env_goofspiel_reward/std": 0.21391310691833496,
-      "sampling/importance_sampling_ratio/max": 1.9750086545944214,
-      "sampling/importance_sampling_ratio/mean": 1.0017020225524902,
-      "sampling/importance_sampling_ratio/min": 0.4652911275625229,
-      "sampling/sampling_logp_difference/max": 0.7616503953933715,
-      "sampling/sampling_logp_difference/mean": 0.048386485874652864,
       "step": 30,
-      "step_time": 5.606105185600154
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -216,29 +216,29 @@
       "completions/clipped_ratio": 0.0,
       "completions/max_length": 374.0,
       "completions/max_terminated_length": 374.0,
-      "completions/mean_length": 290.9440002441406,
-      "completions/mean_terminated_length": 290.9440002441406,
-      "completions/min_length": 212.0,
-      "completions/min_terminated_length": 212.0,
-      "entropy": 0.2889633044600487,
-      "epoch": 0.00875,
-      "frac_reward_zero_std": 0.8479999780654908,
-      "grad_norm": 0.0859375,
-      "kl": 0.22455788254737855,
       "learning_rate": 9.666336e-06,
-      "loss": 6.903421599417925e-05,
-      "num_tokens": 1474071.0,
-      "reward": 0.052800000086426734,
-      "reward_std": 0.06788224689662456,
-      "rewards/env_goofspiel_reward/mean": 0.052800000086426734,
-      "rewards/env_goofspiel_reward/std": 0.16736633479595184,
-      "sampling/importance_sampling_ratio/max": 1.6834668874740601,
-      "sampling/importance_sampling_ratio/mean": 0.9958889007568359,
-      "sampling/importance_sampling_ratio/min": 0.43778104782104493,
-      "sampling/sampling_logp_difference/max": 0.8263626098632812,
-      "sampling/sampling_logp_difference/mean": 0.0358995582908392,
       "step": 35,
-      "step_time": 5.237710429800063
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -247,31 +247,31 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
-      "completions/max_length": 374.0,
-      "completions/max_terminated_length": 374.0,
-      "completions/mean_length": 286.0199951171875,
-      "completions/mean_terminated_length": 286.0199951171875,
       "completions/min_length": 212.0,
       "completions/min_terminated_length": 212.0,
-      "entropy": 0.24612685889005662,
-      "epoch": 0.01,
-      "frac_reward_zero_std": 0.8159999847412109,
-      "grad_norm": 0.32421875,
-      "kl": 0.1633337877690792,
-      "learning_rate": 9.950637941994321e-06,
-      "loss": -2.846699208021164e-05,
-      "num_tokens": 1681137.0,
-      "reward": 0.060000001639127734,
-      "reward_std": 0.08485281020402909,
-      "rewards/env_goofspiel_reward/mean": 0.060000001639127734,
-      "rewards/env_goofspiel_reward/std": 0.19129492342472076,
-      "sampling/importance_sampling_ratio/max": 1.5720022439956665,
-      "sampling/importance_sampling_ratio/mean": 0.9784366488456726,
-      "sampling/importance_sampling_ratio/min": 0.47974646687507627,
-      "sampling/sampling_logp_difference/max": 0.702510929107666,
-      "sampling/sampling_logp_difference/mean": 0.03313286602497101,
       "step": 40,
-      "step_time": 5.238652371199805
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -280,31 +280,31 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
-      "completions/max_length": 660.4,
-      "completions/max_terminated_length": 660.4,
-      "completions/mean_length": 489.1719970703125,
-      "completions/mean_terminated_length": 489.1719970703125,
-      "completions/min_length": 322.4,
-      "completions/min_terminated_length": 322.4,
-      "entropy": 0.3432160258293152,
-      "epoch": 0.01125,
-      "frac_reward_zero_std": 0.7599999785423279,
-      "grad_norm": 0.2255859375,
-      "kl": 0.3002031177282333,
-      "learning_rate": 9.950629581350144e-06,
-      "loss": 0.0002033085562288761,
-      "num_tokens": 1938183.0,
-      "reward": 0.03768000081181526,
-      "reward_std": 0.05008201580494642,
-      "rewards/env_goofspiel_reward/mean": 0.03768000081181526,
-      "rewards/env_goofspiel_reward/std": 0.12525621727108954,
-      "sampling/importance_sampling_ratio/max": 1.7067772150039673,
-      "sampling/importance_sampling_ratio/mean": 1.0024658203125,
-      "sampling/importance_sampling_ratio/min": 0.5572002470493317,
-      "sampling/sampling_logp_difference/max": 0.6115247011184692,
-      "sampling/sampling_logp_difference/mean": 0.03118431307375431,
       "step": 45,
-      "step_time": 7.8333522611999795
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -313,31 +313,31 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
-      "completions/max_length": 731.6,
-      "completions/max_terminated_length": 731.6,
-      "completions/mean_length": 558.283984375,
-      "completions/mean_terminated_length": 558.283984375,
-      "completions/min_length": 408.0,
-      "completions/min_terminated_length": 408.0,
-      "entropy": 0.29817005395889284,
-      "epoch": 0.0125,
-      "frac_reward_zero_std": 0.8399999737739563,
-      "grad_norm": 0.14453125,
-      "kl": 0.14909197837114335,
-      "learning_rate": 9.950614789456512e-06,
-      "loss": -3.2902939710766074e-05,
-      "num_tokens": 2214781.0,
-      "reward": 0.03273333106189966,
-      "reward_std": 0.04195500072091818,
-      "rewards/env_goofspiel_reward/mean": 0.032733332738280295,
-      "rewards/env_goofspiel_reward/std": 0.12340526506304741,
-      "sampling/importance_sampling_ratio/max": 1.7580672979354859,
-      "sampling/importance_sampling_ratio/mean": 1.0121817111968994,
-      "sampling/importance_sampling_ratio/min": 0.5544978082180023,
-      "sampling/sampling_logp_difference/max": 0.6947145342826844,
-      "sampling/sampling_logp_difference/mean": 0.02995888851583004,
       "step": 50,
-      "step_time": 8.399313552200056
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -346,31 +346,31 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
-      "completions/max_length": 737.0,
-      "completions/max_terminated_length": 737.0,
-      "completions/mean_length": 563.09599609375,
-      "completions/mean_terminated_length": 563.09599609375,
-      "completions/min_length": 408.0,
-      "completions/min_terminated_length": 408.0,
-      "entropy": 0.3130415454506874,
-      "epoch": 0.01375,
-      "frac_reward_zero_std": 0.8319999814033509,
-      "grad_norm": 0.12451171875,
-      "kl": 0.17116529867053032,
-      "learning_rate": 9.95059356633892e-06,
-      "loss": 0.00011836685007438064,
-      "num_tokens": 2492933.0,
-      "reward": 0.0414799977093935,
-      "reward_std": 0.04919577315449715,
-      "rewards/env_goofspiel_reward/mean": 0.0414799977093935,
-      "rewards/env_goofspiel_reward/std": 0.13337331116199494,
-      "sampling/importance_sampling_ratio/max": 1.614890694618225,
-      "sampling/importance_sampling_ratio/mean": 0.9976115822792053,
-      "sampling/importance_sampling_ratio/min": 0.5176251292228699,
-      "sampling/sampling_logp_difference/max": 0.5567813754081726,
-      "sampling/sampling_logp_difference/mean": 0.030348184704780578,
       "step": 55,
-      "step_time": 8.462650469800156
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -379,31 +379,31 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
-      "completions/max_length": 731.8,
-      "completions/max_terminated_length": 731.8,
-      "completions/mean_length": 561.4599853515625,
-      "completions/mean_terminated_length": 561.4599853515625,
-      "completions/min_length": 408.0,
-      "completions/min_terminated_length": 408.0,
-      "entropy": 0.3217264384031296,
-      "epoch": 0.015,
-      "frac_reward_zero_std": 0.8239999771118164,
-      "grad_norm": 0.103515625,
-      "kl": 0.16943022534251212,
-      "learning_rate": 9.950565912033946e-06,
-      "loss": 0.00016170007875189186,
-      "num_tokens": 2768999.0,
-      "reward": 0.046799997240304946,
-      "reward_std": 0.06128258481621742,
-      "rewards/env_goofspiel_reward/mean": 0.046799997240304946,
-      "rewards/env_goofspiel_reward/std": 0.1459730952978134,
-      "sampling/importance_sampling_ratio/max": 1.6248928308486938,
-      "sampling/importance_sampling_ratio/mean": 0.994913375377655,
-      "sampling/importance_sampling_ratio/min": 0.5438100695610046,
-      "sampling/sampling_logp_difference/max": 0.5243477821350098,
-      "sampling/sampling_logp_difference/mean": 0.02915378250181675,
       "step": 60,
-      "step_time": 8.311450370799957
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -412,31 +412,31 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
-      "completions/max_length": 936.2,
-      "completions/max_terminated_length": 936.2,
-      "completions/mean_length": 718.1879760742188,
-      "completions/mean_terminated_length": 718.1879760742188,
-      "completions/min_length": 517.2,
-      "completions/min_terminated_length": 517.2,
-      "entropy": 0.3693429589271545,
-      "epoch": 0.01625,
-      "frac_reward_zero_std": 0.8319999694824218,
-      "grad_norm": 0.10400390625,
-      "kl": 0.16856326386332512,
-      "learning_rate": 9.950531826589252e-06,
-      "loss": 9.133372223004698e-05,
-      "num_tokens": 3085009.0,
-      "reward": 0.03046666570007801,
-      "reward_std": 0.04365205764770508,
-      "rewards/env_goofspiel_reward/mean": 0.030466666072607042,
-      "rewards/env_goofspiel_reward/std": 0.1246792048215866,
-      "sampling/importance_sampling_ratio/max": 2.0000662088394163,
-      "sampling/importance_sampling_ratio/mean": 0.9988593339920044,
-      "sampling/importance_sampling_ratio/min": 0.5936131238937378,
-      "sampling/sampling_logp_difference/max": 0.661067008972168,
-      "sampling/sampling_logp_difference/mean": 0.029154302552342415,
       "step": 65,
-      "step_time": 10.795809616799943
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -445,31 +445,31 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
-      "completions/max_length": 1074.6,
-      "completions/max_terminated_length": 1074.6,
-      "completions/mean_length": 809.1799682617187,
-      "completions/mean_terminated_length": 809.1799682617187,
-      "completions/min_length": 590.0,
-      "completions/min_terminated_length": 590.0,
-      "entropy": 0.3258677929639816,
-      "epoch": 0.0175,
-      "frac_reward_zero_std": 0.8239999771118164,
-      "grad_norm": 0.232421875,
-      "kl": 0.1824295423924923,
-      "learning_rate": 9.950491310063582e-06,
-      "loss": 0.00022613720502704382,
-      "num_tokens": 3421737.0,
-      "reward": 0.039453331381082535,
-      "reward_std": 0.048102113604545596,
-      "rewards/env_goofspiel_reward/mean": 0.03945333361625671,
-      "rewards/env_goofspiel_reward/std": 0.12717084884643554,
-      "sampling/importance_sampling_ratio/max": 1.7261427879333495,
-      "sampling/importance_sampling_ratio/mean": 1.0054854035377503,
-      "sampling/importance_sampling_ratio/min": 0.6340943217277527,
-      "sampling/sampling_logp_difference/max": 0.4532318115234375,
-      "sampling/sampling_logp_difference/mean": 0.02659378871321678,
       "step": 70,
-      "step_time": 12.004160757599857
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -478,69 +478,69 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
-      "completions/max_length": 1074.0,
-      "completions/max_terminated_length": 1074.0,
-      "completions/mean_length": 813.3439697265625,
-      "completions/mean_terminated_length": 813.3439697265625,
-      "completions/min_length": 590.0,
-      "completions/min_terminated_length": 590.0,
-      "entropy": 0.29610189199447634,
-      "epoch": 0.01875,
-      "frac_reward_zero_std": 0.8639999747276306,
-      "grad_norm": 0.1201171875,
-      "kl": 0.17389494478702544,
-      "learning_rate": 9.950444362526773e-06,
-      "loss": -2.63441979768686e-06,
-      "num_tokens": 3760662.0,
-      "reward": 0.03491999926045537,
-      "reward_std": 0.041691013239324094,
-      "rewards/env_goofspiel_reward/mean": 0.03492000075057149,
-      "rewards/env_goofspiel_reward/std": 0.12171182483434677,
-      "sampling/importance_sampling_ratio/max": 1.5990596771240235,
-      "sampling/importance_sampling_ratio/mean": 1.0042136073112489,
-      "sampling/importance_sampling_ratio/min": 0.5866354703903198,
-      "sampling/sampling_logp_difference/max": 0.49382131099700927,
-      "sampling/sampling_logp_difference/mean": 0.024137656763195993,
       "step": 75,
-      "step_time": 11.794954787200004
     },
     {
-      "epoch": 0.01875,
       "eval_clip_ratio/high_max": 0.0,
       "eval_clip_ratio/high_mean": 0.0,
       "eval_clip_ratio/low_mean": 0.0,
       "eval_clip_ratio/low_min": 0.0,
       "eval_clip_ratio/region_mean": 0.0,
       "eval_completions/clipped_ratio": 0.0,
-      "eval_completions/max_length": 895.0,
-      "eval_completions/max_terminated_length": 895.0,
-      "eval_completions/mean_length": 806.55,
-      "eval_completions/mean_terminated_length": 806.55,
-      "eval_completions/min_length": 725.8,
-      "eval_completions/min_terminated_length": 725.8,
-      "eval_entropy": 0.3078053116798401,
       "eval_frac_reward_zero_std": 0.6,
-      "eval_kl": 0.5635438948869705,
-      "eval_loss": 0.0008250019163824618,
-      "eval_num_tokens": 3760662.0,
-      "eval_reward": 0.06899999883025884,
-      "eval_reward_std": 0.10040915999561548,
-      "eval_rewards/env_goofspiel_reward/mean": 0.06899999883025884,
-      "eval_rewards/env_goofspiel_reward/std": 0.11241451688110829,
-      "eval_runtime": 3.5242,
-      "eval_samples_per_second": 2.838,
-      "eval_sampling/importance_sampling_ratio/max": 1.2238447427749635,
-      "eval_sampling/importance_sampling_ratio/mean": 0.9734418511390686,
-      "eval_sampling/importance_sampling_ratio/min": 0.7944899320602417,
-      "eval_sampling/sampling_logp_difference/max": 0.2762513041496277,
-      "eval_sampling/sampling_logp_difference/mean": 0.028397564217448233,
-      "eval_steps_per_second": 0.851,
       "step": 75
     }
   ],
   "logging_steps": 5,
-  "max_steps": 12000,
-  "num_input_tokens_seen": 3760662,
   "num_train_epochs": 3,
   "save_steps": 500,
   "stateful_callbacks": {
@@ -556,7 +556,7 @@
     }
   },
   "total_flos": 0.0,
-  "train_batch_size": 25,
   "trial_name": null,
   "trial_params": null
 }

   "best_global_step": null,
   "best_metric": null,
   "best_model_checkpoint": null,
+  "epoch": 0.009000360014400577,
   "eval_steps": 500,
   "global_step": 75,
   "is_hyper_param_search": false,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
+      "completions/max_length": 365.6,
+      "completions/max_terminated_length": 365.6,
+      "completions/mean_length": 292.2166809082031,
+      "completions/mean_terminated_length": 292.2166809082031,
+      "completions/min_length": 174.8,
+      "completions/min_terminated_length": 174.8,
+      "entropy": 0.7345801413059234,
+      "epoch": 0.0006000240009600384,
+      "frac_reward_zero_std": 0.650000023841858,
+      "grad_norm": 0.578125,
+      "kl": 0.014322867337614297,
       "learning_rate": 1.137216e-06,
+      "loss": 0.00023176579270511866,
+      "num_tokens": 101638.0,
+      "reward": 0.023166669206693767,
+      "reward_std": 0.03771236310712993,
+      "rewards/env_goofspiel_reward/mean": 0.023166668484918773,
+      "rewards/env_goofspiel_reward/std": 0.10488454704172909,
+      "sampling/importance_sampling_ratio/max": 2.230660581588745,
+      "sampling/importance_sampling_ratio/mean": 1.0512551069259644,
+      "sampling/importance_sampling_ratio/min": 0.45719883143901824,
+      "sampling/sampling_logp_difference/max": 0.9388755321502685,
+      "sampling/sampling_logp_difference/mean": 0.08014384806156158,
       "step": 5,
+      "step_time": 2.8670244561999425
     },
     {
       "clip_ratio/high_max": 0.0,
       "completions/clipped_ratio": 0.0,
       "completions/max_length": 373.8,
       "completions/max_terminated_length": 373.8,
+      "completions/mean_length": 284.3750061035156,
+      "completions/mean_terminated_length": 284.3750061035156,
+      "completions/min_length": 197.4,
+      "completions/min_terminated_length": 197.4,
+      "entropy": 0.7396668076515198,
+      "epoch": 0.0012000480019200767,
+      "frac_reward_zero_std": 0.5333333492279053,
+      "grad_norm": 0.6484375,
+      "kl": 0.008018274139612914,
       "learning_rate": 2.5587359999999995e-06,
+      "loss": 0.0010880917310714723,
+      "num_tokens": 201985.0,
+      "reward": 0.1135000076610595,
+      "reward_std": 0.1355288046877831,
+      "rewards/env_goofspiel_reward/mean": 0.11350000470411033,
+      "rewards/env_goofspiel_reward/std": 0.24339603506959975,
+      "sampling/importance_sampling_ratio/max": 1.9386430501937866,
+      "sampling/importance_sampling_ratio/mean": 1.0104641795158387,
+      "sampling/importance_sampling_ratio/min": 0.48966296911239626,
+      "sampling/sampling_logp_difference/max": 0.7527793884277344,
+      "sampling/sampling_logp_difference/mean": 0.06743223667144775,
       "step": 10,
+      "step_time": 2.562607514999763
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
+      "completions/max_length": 373.8,
+      "completions/max_terminated_length": 373.8,
+      "completions/mean_length": 292.29168090820315,
+      "completions/mean_terminated_length": 292.29168090820315,
+      "completions/min_length": 205.4,
+      "completions/min_terminated_length": 205.4,
+      "entropy": 0.6868447959423065,
+      "epoch": 0.0018000720028801152,
+      "frac_reward_zero_std": 0.7666666865348816,
+      "grad_norm": 0.248046875,
+      "kl": 0.015538515662774444,
       "learning_rate": 3.9802559999999995e-06,
+      "loss": -0.0002336445264518261,
+      "num_tokens": 303718.0,
+      "reward": 0.02416666953358799,
+      "reward_std": 0.03653385282959789,
+      "rewards/env_goofspiel_reward/mean": 0.024166667682584374,
+      "rewards/env_goofspiel_reward/std": 0.10811053770594299,
+      "sampling/importance_sampling_ratio/max": 1.5562421321868896,
+      "sampling/importance_sampling_ratio/mean": 0.9962499618530274,
+      "sampling/importance_sampling_ratio/min": 0.4864930033683777,
+      "sampling/sampling_logp_difference/max": 0.7846987843513489,
+      "sampling/sampling_logp_difference/mean": 0.05959557741880417,
       "step": 15,
+      "step_time": 2.4648701715999777
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
+      "completions/max_length": 365.4,
+      "completions/max_terminated_length": 365.4,
+      "completions/mean_length": 286.4166748046875,
+      "completions/mean_terminated_length": 286.4166748046875,
       "completions/min_length": 199.6,
       "completions/min_terminated_length": 199.6,
+      "entropy": 0.6791316926479339,
+      "epoch": 0.0024000960038401535,
+      "frac_reward_zero_std": 0.7833333492279053,
+      "grad_norm": 0.419921875,
+      "kl": 0.03971561994403601,
       "learning_rate": 5.401775999999999e-06,
+      "loss": 0.00017259303713217377,
+      "num_tokens": 403266.0,
+      "reward": 0.05466667115688324,
+      "reward_std": 0.07825315818190574,
+      "rewards/env_goofspiel_reward/mean": 0.054666668176651,
+      "rewards/env_goofspiel_reward/std": 0.1692157879471779,
+      "sampling/importance_sampling_ratio/max": 1.7059913635253907,
+      "sampling/importance_sampling_ratio/mean": 1.0254743337631225,
+      "sampling/importance_sampling_ratio/min": 0.5575755715370179,
+      "sampling/sampling_logp_difference/max": 0.5765037894248962,
+      "sampling/sampling_logp_difference/mean": 0.06591257303953171,
       "step": 20,
+      "step_time": 2.4083536974001616
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
+      "completions/max_length": 373.6,
+      "completions/max_terminated_length": 373.6,
+      "completions/mean_length": 279.07501831054685,
+      "completions/mean_terminated_length": 279.07501831054685,
+      "completions/min_length": 206.8,
+      "completions/min_terminated_length": 206.8,
+      "entropy": 0.5845985025167465,
+      "epoch": 0.003000120004800192,
+      "frac_reward_zero_std": 0.8666666865348815,
+      "grad_norm": 0.2080078125,
+      "kl": 0.07888290733098983,
       "learning_rate": 6.8232959999999994e-06,
+      "loss": 0.00014582456788048148,
+      "num_tokens": 501949.0,
+      "reward": 0.029750002920627593,
+      "reward_std": 0.042544259876012805,
+      "rewards/env_goofspiel_reward/mean": 0.029750000685453415,
+      "rewards/env_goofspiel_reward/std": 0.11651719957590104,
+      "sampling/importance_sampling_ratio/max": 1.5395583629608154,
+      "sampling/importance_sampling_ratio/mean": 0.9871565222740173,
+      "sampling/importance_sampling_ratio/min": 0.6314660668373108,
+      "sampling/sampling_logp_difference/max": 0.43876824378967283,
+      "sampling/sampling_logp_difference/mean": 0.04884573593735695,
       "step": 25,
+      "step_time": 2.4319384599999467
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
+      "completions/max_length": 373.6,
+      "completions/max_terminated_length": 373.6,
+      "completions/mean_length": 294.0166748046875,
+      "completions/mean_terminated_length": 294.0166748046875,
       "completions/min_length": 212.0,
       "completions/min_terminated_length": 212.0,
+      "entropy": 0.5196643978357315,
+      "epoch": 0.0036001440057602304,
+      "frac_reward_zero_std": 0.850000011920929,
+      "grad_norm": 0.62109375,
+      "kl": 0.1175543449819088,
       "learning_rate": 8.244816e-06,
+      "loss": 0.0001264215330593288,
+      "num_tokens": 604591.0,
+      "reward": 0.044750004261732104,
+      "reward_std": 0.04985102787613869,
+      "rewards/env_goofspiel_reward/mean": 0.044750002399086955,
+      "rewards/env_goofspiel_reward/std": 0.125959412753582,
+      "sampling/importance_sampling_ratio/max": 1.7827884435653687,
+      "sampling/importance_sampling_ratio/mean": 0.9959828734397889,
+      "sampling/importance_sampling_ratio/min": 0.6090725898742676,
+      "sampling/sampling_logp_difference/max": 0.5527279376983643,
+      "sampling/sampling_logp_difference/mean": 0.05205402001738548,
       "step": 30,
+      "step_time": 2.3975842315998306
     },
     {
       "clip_ratio/high_max": 0.0,
       "completions/clipped_ratio": 0.0,
       "completions/max_length": 374.0,
       "completions/max_terminated_length": 374.0,
+      "completions/mean_length": 291.40834350585936,
+      "completions/mean_terminated_length": 291.40834350585936,
+      "completions/min_length": 219.2,
+      "completions/min_terminated_length": 219.2,
+      "entropy": 0.4317367374897003,
+      "epoch": 0.004200168006720269,
+      "frac_reward_zero_std": 0.7500000119209289,
+      "grad_norm": 0.00531005859375,
+      "kl": 0.12076274678111076,
       "learning_rate": 9.666336e-06,
+      "loss": 0.0003318458097055554,
+      "num_tokens": 706062.0,
+      "reward": 0.09500000476837159,
+      "reward_std": 0.12020815908908844,
+      "rewards/env_goofspiel_reward/mean": 0.09500000327825546,
+      "rewards/env_goofspiel_reward/std": 0.21089642345905305,
+      "sampling/importance_sampling_ratio/max": 1.4750993490219115,
+      "sampling/importance_sampling_ratio/mean": 0.9911892533302307,
+      "sampling/importance_sampling_ratio/min": 0.5305798888206482,
+      "sampling/sampling_logp_difference/max": 0.7321011543273925,
+      "sampling/sampling_logp_difference/mean": 0.043305123969912526,
       "step": 35,
+      "step_time": 2.43143495660006
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
+      "completions/max_length": 373.4,
+      "completions/max_terminated_length": 373.4,
+      "completions/mean_length": 288.1250061035156,
+      "completions/mean_terminated_length": 288.1250061035156,
       "completions/min_length": 212.0,
       "completions/min_terminated_length": 212.0,
+      "entropy": 0.351773801445961,
+      "epoch": 0.004800192007680307,
+      "frac_reward_zero_std": 0.8666666865348815,
+      "grad_norm": 0.33203125,
+      "kl": 0.05839128475636244,
+      "learning_rate": 9.950639527236806e-06,
+      "loss": 4.926343681290746e-05,
+      "num_tokens": 806862.0,
+      "reward": 0.040000003576278684,
+      "reward_std": 0.05656854510307312,
+      "rewards/env_goofspiel_reward/mean": 0.04000000059604645,
+      "rewards/env_goofspiel_reward/std": 0.11344237923622132,
+      "sampling/importance_sampling_ratio/max": 1.5833612918853759,
+      "sampling/importance_sampling_ratio/mean": 1.0104422450065613,
+      "sampling/importance_sampling_ratio/min": 0.6002241253852845,
+      "sampling/sampling_logp_difference/max": 0.5244450092315673,
+      "sampling/sampling_logp_difference/mean": 0.03759892582893372,
       "step": 40,
+      "step_time": 2.415808950400242
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
+      "completions/max_length": 373.8,
+      "completions/max_terminated_length": 373.8,
+      "completions/mean_length": 291.74168090820314,
+      "completions/mean_terminated_length": 291.74168090820314,
+      "completions/min_length": 212.0,
+      "completions/min_terminated_length": 212.0,
+      "entropy": 0.3240418329834938,
+      "epoch": 0.005400216008640346,
+      "frac_reward_zero_std": 0.9166666865348816,
+      "grad_norm": 0.0439453125,
+      "kl": 0.13300706073641777,
+      "learning_rate": 9.950637606636539e-06,
+      "loss": 0.0001355916727334261,
+      "num_tokens": 907008.0,
+      "reward": 0.034833335876464845,
+      "reward_std": 0.03535534143447876,
+      "rewards/env_goofspiel_reward/mean": 0.03483333364129067,
+      "rewards/env_goofspiel_reward/std": 0.14086373001337052,
+      "sampling/importance_sampling_ratio/max": 1.4610427141189575,
+      "sampling/importance_sampling_ratio/mean": 0.9817873358726501,
+      "sampling/importance_sampling_ratio/min": 0.6326651930809021,
+      "sampling/sampling_logp_difference/max": 0.46174774169921873,
+      "sampling/sampling_logp_difference/mean": 0.040712539479136466,
       "step": 45,
+      "step_time": 2.417739940000138
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
+      "completions/max_length": 365.0,
+      "completions/max_terminated_length": 365.0,
+      "completions/mean_length": 279.3333435058594,
+      "completions/mean_terminated_length": 279.3333435058594,
+      "completions/min_length": 212.0,
+      "completions/min_terminated_length": 212.0,
+      "entropy": 0.3186733976006508,
+      "epoch": 0.006000240009600384,
+      "frac_reward_zero_std": 0.8833333611488342,
+      "grad_norm": 0.353515625,
+      "kl": 0.06966875828802585,
+      "learning_rate": 9.950634208652256e-06,
+      "loss": 0.00012671776348724962,
+      "num_tokens": 1005578.0,
+      "reward": 0.034916669048834593,
+      "reward_std": 0.049615327350329606,
+      "rewards/env_goofspiel_reward/mean": 0.03491666756453924,
+      "rewards/env_goofspiel_reward/std": 0.13708889302797617,
+      "sampling/importance_sampling_ratio/max": 1.5069894075393677,
+      "sampling/importance_sampling_ratio/mean": 1.0077889800071715,
+      "sampling/importance_sampling_ratio/min": 0.7747546076774597,
+      "sampling/sampling_logp_difference/max": 0.3697906732559204,
+      "sampling/sampling_logp_difference/mean": 0.03059108220040798,
       "step": 50,
+      "step_time": 2.3773288868003872
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
+      "completions/max_length": 374.0,
+      "completions/max_terminated_length": 374.0,
+      "completions/mean_length": 305.49168090820314,
+      "completions/mean_terminated_length": 305.49168090820314,
+      "completions/min_length": 212.0,
+      "completions/min_terminated_length": 212.0,
+      "entropy": 0.29847966730594633,
+      "epoch": 0.006600264010560422,
+      "frac_reward_zero_std": 0.900000023841858,
+      "grad_norm": 0.30078125,
+      "kl": 0.08750866688787937,
+      "learning_rate": 9.950629333285305e-06,
+      "loss": -2.145505277439952e-06,
+      "num_tokens": 1110455.0,
+      "reward": 0.035000003129243853,
+      "reward_std": 0.04949747696518898,
+      "rewards/env_goofspiel_reward/mean": 0.03500000052154064,
+      "rewards/env_goofspiel_reward/std": 0.1412438616156578,
+      "sampling/importance_sampling_ratio/max": 1.3705053567886352,
+      "sampling/importance_sampling_ratio/mean": 1.0030481100082398,
+      "sampling/importance_sampling_ratio/min": 0.6937733888626099,
+      "sampling/sampling_logp_difference/max": 0.4317422866821289,
+      "sampling/sampling_logp_difference/mean": 0.03450411073863506,
       "step": 55,
+      "step_time": 2.4346962132000045
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
+      "completions/max_length": 374.0,
+      "completions/max_terminated_length": 374.0,
+      "completions/mean_length": 294.62501220703126,
+      "completions/mean_terminated_length": 294.62501220703126,
+      "completions/min_length": 212.0,
+      "completions/min_terminated_length": 212.0,
+      "entropy": 0.2199392184615135,
+      "epoch": 0.007200288011520461,
+      "frac_reward_zero_std": 0.7666666746139527,
+      "grad_norm": 0.1328125,
+      "kl": 0.20852462351322174,
+      "learning_rate": 9.950622980537618e-06,
+      "loss": -1.4243402983993291e-05,
+      "num_tokens": 1211770.0,
+      "reward": 0.08483333513140678,
+      "reward_std": 0.09215958416461945,
+      "rewards/env_goofspiel_reward/mean": 0.08483333475887775,
+      "rewards/env_goofspiel_reward/std": 0.2297523573040962,
+      "sampling/importance_sampling_ratio/max": 1.3913918256759643,
+      "sampling/importance_sampling_ratio/mean": 0.9923399925231934,
+      "sampling/importance_sampling_ratio/min": 0.7218815922737122,
+      "sampling/sampling_logp_difference/max": 0.3741787910461426,
+      "sampling/sampling_logp_difference/mean": 0.023421294614672662,
       "step": 60,
+      "step_time": 2.4339242404003016
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
+      "completions/max_length": 365.0,
+      "completions/max_terminated_length": 365.0,
+      "completions/mean_length": 299.108349609375,
+      "completions/mean_terminated_length": 299.108349609375,
+      "completions/min_length": 212.0,
+      "completions/min_terminated_length": 212.0,
+      "entropy": 0.14547686353325845,
+      "epoch": 0.0078003120124804995,
+      "frac_reward_zero_std": 0.8666666865348815,
+      "grad_norm": 0.01318359375,
+      "kl": 1.5776881486177445,
+      "learning_rate": 9.950615150411705e-06,
+      "loss": 0.00020953675266355276,
+      "num_tokens": 1315125.0,
+      "reward": 0.03991667032241821,
+      "reward_std": 0.05668639615178108,
+      "rewards/env_goofspiel_reward/mean": 0.03991666734218598,
+      "rewards/env_goofspiel_reward/std": 0.15578595399856568,
+      "sampling/importance_sampling_ratio/max": 1.5614466190338134,
+      "sampling/importance_sampling_ratio/mean": 1.0267313480377198,
+      "sampling/importance_sampling_ratio/min": 0.8062139034271241,
+      "sampling/sampling_logp_difference/max": 0.38132710456848146,
+      "sampling/sampling_logp_difference/mean": 0.017732756957411767,
       "step": 65,
+      "step_time": 2.3855804428001646
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
+      "completions/max_length": 373.4,
+      "completions/max_terminated_length": 373.4,
+      "completions/mean_length": 288.8666748046875,
+      "completions/mean_terminated_length": 288.8666748046875,
+      "completions/min_length": 212.0,
+      "completions/min_terminated_length": 212.0,
+      "entropy": 0.36806915551424024,
+      "epoch": 0.008400336013440538,
+      "frac_reward_zero_std": 0.8333333730697632,
+      "grad_norm": 0.103515625,
+      "kl": 0.154165069013834,
+      "learning_rate": 9.950605842910668e-06,
+      "loss": 3.9057480171322824e-05,
+      "num_tokens": 1415706.0,
+      "reward": 0.0650000050663948,
+      "reward_std": 0.07778174877166748,
+      "rewards/env_goofspiel_reward/mean": 0.06500000134110451,
+      "rewards/env_goofspiel_reward/std": 0.20113323032855987,
+      "sampling/importance_sampling_ratio/max": 1.3685919523239136,
+      "sampling/importance_sampling_ratio/mean": 0.9923707365989685,
+      "sampling/importance_sampling_ratio/min": 0.6977368593215942,
+      "sampling/sampling_logp_difference/max": 0.37837958335876465,
+      "sampling/sampling_logp_difference/mean": 0.023688069358468056,
       "step": 70,
+      "step_time": 2.419219413600149
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": 0.0,
+      "completions/max_length": 365.0,
+      "completions/max_terminated_length": 365.0,
+      "completions/mean_length": 296.4750122070312,
+      "completions/mean_terminated_length": 296.4750122070312,
+      "completions/min_length": 218.8,
+      "completions/min_terminated_length": 218.8,
+      "entropy": 0.44495113492012023,
+      "epoch": 0.009000360014400577,
+      "frac_reward_zero_std": 0.7666666984558106,
+      "grad_norm": 0.13671875,
+      "kl": 0.308458948135376,
+      "learning_rate": 9.950595058038197e-06,
+      "loss": 0.00013219380052760245,
+      "num_tokens": 1517756.0,
+      "reward": 0.054750004410743715,
+      "reward_std": 0.07813530415296555,
+      "rewards/env_goofspiel_reward/mean": 0.05475000143051147,
+      "rewards/env_goofspiel_reward/std": 0.17614837288856505,
+      "sampling/importance_sampling_ratio/max": 1.4038194417953491,
+      "sampling/importance_sampling_ratio/mean": 0.9863661766052246,
+      "sampling/importance_sampling_ratio/min": 0.6411556363105774,
+      "sampling/sampling_logp_difference/max": 0.5436622142791748,
+      "sampling/sampling_logp_difference/mean": 0.029225154593586922,
       "step": 75,
+      "step_time": 2.370857671000158
     },
     {
+      "epoch": 0.009000360014400577,
       "eval_clip_ratio/high_max": 0.0,
       "eval_clip_ratio/high_mean": 0.0,
       "eval_clip_ratio/low_mean": 0.0,
       "eval_clip_ratio/low_min": 0.0,
       "eval_clip_ratio/region_mean": 0.0,
       "eval_completions/clipped_ratio": 0.0,
+      "eval_completions/max_length": 274.2,
+      "eval_completions/max_terminated_length": 274.2,
+      "eval_completions/mean_length": 254.7,
+      "eval_completions/mean_terminated_length": 254.7,
+      "eval_completions/min_length": 235.4,
+      "eval_completions/min_terminated_length": 235.4,
+      "eval_entropy": 0.4275285005569458,
       "eval_frac_reward_zero_std": 0.6,
+      "eval_kl": 0.317794269323349,
+      "eval_loss": 9.086851787287742e-05,
+      "eval_num_tokens": 1517756.0,
+      "eval_reward": 0.12000000476837158,
+      "eval_reward_std": 0.16970562934875488,
+      "eval_rewards/env_goofspiel_reward/mean": 0.12000000476837158,
+      "eval_rewards/env_goofspiel_reward/std": 0.24000003337860107,
+      "eval_runtime": 1.6574,
+      "eval_samples_per_second": 6.034,
+      "eval_sampling/importance_sampling_ratio/max": 1.1251073837280274,
+      "eval_sampling/importance_sampling_ratio/mean": 0.9802677392959595,
+      "eval_sampling/importance_sampling_ratio/min": 0.8105595707893372,
+      "eval_sampling/sampling_logp_difference/max": 0.2066459536552429,
+      "eval_sampling/sampling_logp_difference/mean": 0.03130748393014073,
+      "eval_steps_per_second": 1.81,
       "step": 75
     }
   ],
   "logging_steps": 5,
+  "max_steps": 24999,
+  "num_input_tokens_seen": 1517756,
   "num_train_epochs": 3,
   "save_steps": 500,
   "stateful_callbacks": {
     }
   },
   "total_flos": 0.0,
+  "train_batch_size": 12,
   "trial_name": null,
   "trial_params": null
 }

training_args.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8c53ec641d35398511966f6bbf25ac001ca90c58e99dcc4975f403287921d176
 size 7185

 version https://git-lfs.github.com/spec/v1
+oid sha256:d6a4c97c4543543d2fa415f06995465636d6f177fd18bba16e72de2f4e49a937
 size 7185