Instructions to use Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Jordansky/augmented-619958b5bf46bea2")
model = PeftModel.from_pretrained(base_model, "Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil")

Transformers

How to use Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil

SGLang

How to use Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil with Docker Model Runner:
```
docker model run hf.co/Gege24/dejavu-ginrummy-intercode-2507-newcode-dancil
```

Gege24 commited on May 31

Commit

c3cbdd2

verified ·

1 Parent(s): 6228505

Upload checkpoint from task 1

Browse files

Files changed (5) hide show

adapter_config.json +4 -4
adapter_model.safetensors +1 -1
loss.txt +1 -1
trainer_state.json +379 -424
training_args.bin +1 -1

adapter_config.json CHANGED Viewed

@@ -29,13 +29,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
     "q_proj",
-    "up_proj",
-    "v_proj",
     "o_proj",
-    "k_proj",
     "down_proj",
-    "gate_proj"
   ],
   "target_parameters": null,
   "task_type": "CAUSAL_LM",

   "rank_pattern": {},
   "revision": null,
   "target_modules": [
+    "k_proj",
     "q_proj",
     "o_proj",
     "down_proj",
+    "up_proj",
+    "gate_proj",
+    "v_proj"
   ],
   "target_parameters": null,
   "task_type": "CAUSAL_LM",

adapter_model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:39224a614d588f842f93da0fe21a6c4b78dd20343673a1920303f0c5aacdd1de
 size 239536272

 version https://git-lfs.github.com/spec/v1
+oid sha256:4c54843db773c25a827d2d13d02d4a752008596f1bf18d861d2c05a49cc5e126
 size 239536272

loss.txt CHANGED Viewed

	@@ -1 +1 @@
1	- ~~340~~,no_eval


1	+ 316,no_eval

trainer_state.json CHANGED Viewed

@@ -2,630 +2,585 @@
   "best_global_step": null,
   "best_metric": null,
   "best_model_checkpoint": null,
-  "epoch": 3.953488372093023,
   "eval_steps": 500,
-  "global_step": 340,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
   "log_history": [
     {
-      "epoch": 0.05813953488372093,
-      "grad_norm": 3.3227427005767822,
       "learning_rate": 1.3636363636363635e-05,
-      "loss": 1.4688,
-      "mean_token_accuracy": 0.681653356552124,
-      "num_tokens": 908617.0,
       "step": 5
     },
     {
-      "epoch": 0.11627906976744186,
-      "grad_norm": 0.8916869759559631,
       "learning_rate": 3.068181818181818e-05,
-      "loss": 1.3353,
-      "mean_token_accuracy": 0.693474531173706,
-      "num_tokens": 1787376.0,
       "step": 10
     },
     {
-      "epoch": 0.1744186046511628,
-      "grad_norm": 0.5437080264091492,
       "learning_rate": 4.7727272727272724e-05,
-      "loss": 1.1146,
-      "mean_token_accuracy": 0.7165812611579895,
-      "num_tokens": 2673208.0,
       "step": 15
     },
     {
-      "epoch": 0.23255813953488372,
-      "grad_norm": 0.441089004278183,
       "learning_rate": 6.477272727272727e-05,
-      "loss": 0.9604,
-      "mean_token_accuracy": 0.7472989082336425,
-      "num_tokens": 3579185.0,
       "step": 20
     },
     {
-      "epoch": 0.29069767441860467,
-      "grad_norm": 0.3763512372970581,
-      "learning_rate": 7.4997725098906e-05,
-      "loss": 0.9084,
-      "mean_token_accuracy": 0.7577781558036805,
-      "num_tokens": 4407100.0,
       "step": 25
     },
     {
-      "epoch": 0.3488372093023256,
-      "grad_norm": 0.3838900327682495,
-      "learning_rate": 7.497213668780336e-05,
-      "loss": 0.7685,
-      "mean_token_accuracy": 0.7860511183738709,
-      "num_tokens": 5296594.0,
       "step": 30
     },
     {
-      "epoch": 0.4069767441860465,
-      "grad_norm": 0.3650754988193512,
-      "learning_rate": 7.491814219525521e-05,
-      "loss": 0.7932,
-      "mean_token_accuracy": 0.7775424718856812,
-      "num_tokens": 6169145.0,
       "step": 35
     },
     {
-      "epoch": 0.46511627906976744,
-      "grad_norm": 0.3387715518474579,
-      "learning_rate": 7.483579620945575e-05,
-      "loss": 0.6983,
-      "mean_token_accuracy": 0.8077743411064148,
-      "num_tokens": 7089714.0,
       "step": 40
     },
     {
-      "epoch": 0.5232558139534884,
-      "grad_norm": 0.3948509991168976,
-      "learning_rate": 7.472518198183393e-05,
-      "loss": 0.6689,
-      "mean_token_accuracy": 0.8084866642951966,
-      "num_tokens": 7960840.0,
       "step": 45
     },
     {
-      "epoch": 0.5813953488372093,
-      "grad_norm": 0.33756810426712036,
-      "learning_rate": 7.458641134288672e-05,
-      "loss": 0.6744,
-      "mean_token_accuracy": 0.8019166707992553,
-      "num_tokens": 8876139.0,
       "step": 50
     },
     {
-      "epoch": 0.6395348837209303,
-      "grad_norm": 0.3875386118888855,
-      "learning_rate": 7.441962458911889e-05,
-      "loss": 0.6292,
-      "mean_token_accuracy": 0.8159096598625183,
-      "num_tokens": 9749095.0,
       "step": 55
     },
     {
-      "epoch": 0.6976744186046512,
-      "grad_norm": 0.4815349876880646,
-      "learning_rate": 7.422499034120398e-05,
-      "loss": 0.5822,
-      "mean_token_accuracy": 0.827684223651886,
-      "num_tokens": 10610501.0,
       "step": 60
     },
     {
-      "epoch": 0.7558139534883721,
-      "grad_norm": 0.43496716022491455,
-      "learning_rate": 7.400270537350939e-05,
-      "loss": 0.6027,
-      "mean_token_accuracy": 0.8274801731109619,
-      "num_tokens": 11478956.0,
       "step": 65
     },
     {
-      "epoch": 0.813953488372093,
-      "grad_norm": 0.3981734812259674,
-      "learning_rate": 7.375299441515847e-05,
-      "loss": 0.5875,
-      "mean_token_accuracy": 0.8244443655014038,
-      "num_tokens": 12434088.0,
       "step": 70
     },
     {
-      "epoch": 0.872093023255814,
-      "grad_norm": 0.45824673771858215,
-      "learning_rate": 7.347610992283035e-05,
-      "loss": 0.5896,
-      "mean_token_accuracy": 0.8241708278656006,
-      "num_tokens": 13289313.0,
       "step": 75
     },
     {
-      "epoch": 0.9302325581395349,
-      "grad_norm": 0.47565343976020813,
-      "learning_rate": 7.317233182552728e-05,
-      "loss": 0.5606,
-      "mean_token_accuracy": 0.8319092392921448,
-      "num_tokens": 14176300.0,
       "step": 80
     },
     {
-      "epoch": 0.9883720930232558,
-      "grad_norm": 0.6352036595344543,
-      "learning_rate": 7.284196724156777e-05,
-      "loss": 0.549,
-      "mean_token_accuracy": 0.833467161655426,
-      "num_tokens": 15039331.0,
       "step": 85
     },
     {
-      "epoch": 1.0465116279069768,
-      "grad_norm": 0.4878791868686676,
-      "learning_rate": 7.248535016809117e-05,
-      "loss": 0.5266,
-      "mean_token_accuracy": 0.838879406452179,
-      "num_tokens": 15951728.0,
       "step": 90
     },
     {
-      "epoch": 1.1046511627906976,
-      "grad_norm": 0.4694942831993103,
-      "learning_rate": 7.210284114338817e-05,
-      "loss": 0.5094,
-      "mean_token_accuracy": 0.8451227068901062,
-      "num_tokens": 16854757.0,
       "step": 95
     },
     {
-      "epoch": 1.1627906976744187,
-      "grad_norm": 0.5654356479644775,
-      "learning_rate": 7.169482688239809e-05,
-      "loss": 0.4883,
-      "mean_token_accuracy": 0.8548959255218506,
-      "num_tokens": 17710044.0,
       "step": 100
     },
     {
-      "epoch": 1.2209302325581395,
-      "grad_norm": 0.5400692224502563,
-      "learning_rate": 7.126171988574178e-05,
-      "loss": 0.4735,
-      "mean_token_accuracy": 0.8540786027908325,
-      "num_tokens": 18583272.0,
       "step": 105
     },
     {
-      "epoch": 1.2790697674418605,
-      "grad_norm": 0.4803585112094879,
-      "learning_rate": 7.080395802268541e-05,
-      "loss": 0.4838,
-      "mean_token_accuracy": 0.8504316091537476,
-      "num_tokens": 19479917.0,
       "step": 110
     },
     {
-      "epoch": 1.3372093023255813,
-      "grad_norm": 0.5055525898933411,
-      "learning_rate": 7.032200408845642e-05,
-      "loss": 0.4719,
-      "mean_token_accuracy": 0.85212082862854,
-      "num_tokens": 20428580.0,
       "step": 115
     },
     {
-      "epoch": 1.3953488372093024,
-      "grad_norm": 0.4785888195037842,
-      "learning_rate": 6.981634533635966e-05,
-      "loss": 0.4994,
-      "mean_token_accuracy": 0.8442901730537414,
-      "num_tokens": 21312035.0,
       "step": 120
     },
     {
-      "epoch": 1.4534883720930232,
-      "grad_norm": 0.5483599901199341,
-      "learning_rate": 6.928749298516651e-05,
-      "loss": 0.4768,
-      "mean_token_accuracy": 0.8488313317298889,
-      "num_tokens": 22198657.0,
       "step": 125
     },
     {
-      "epoch": 1.5116279069767442,
-      "grad_norm": 0.44508588314056396,
-      "learning_rate": 6.873598170227484e-05,
-      "loss": 0.4623,
-      "mean_token_accuracy": 0.8573606014251709,
-      "num_tokens": 23065110.0,
       "step": 130
     },
     {
-      "epoch": 1.5697674418604652,
-      "grad_norm": 0.49224478006362915,
-      "learning_rate": 6.816236906316267e-05,
-      "loss": 0.4661,
-      "mean_token_accuracy": 0.8557985186576843,
-      "num_tokens": 23996595.0,
       "step": 135
     },
     {
-      "epoch": 1.627906976744186,
-      "grad_norm": 0.5778976678848267,
-      "learning_rate": 6.756723498768182e-05,
-      "loss": 0.4473,
-      "mean_token_accuracy": 0.8574177622795105,
-      "num_tokens": 24922107.0,
       "step": 140
     },
     {
-      "epoch": 1.6860465116279069,
-      "grad_norm": 0.5565524101257324,
-      "learning_rate": 6.695118115376152e-05,
-      "loss": 0.452,
-      "mean_token_accuracy": 0.860586965084076,
-      "num_tokens": 25764329.0,
       "step": 145
     },
     {
-      "epoch": 1.744186046511628,
-      "grad_norm": 0.5819323062896729,
-      "learning_rate": 6.631483038911461e-05,
-      "loss": 0.4172,
-      "mean_token_accuracy": 0.8694151997566223,
-      "num_tokens": 26602076.0,
       "step": 150
     },
     {
-      "epoch": 1.802325581395349,
-      "grad_norm": 0.5286712050437927,
-      "learning_rate": 6.565882604156153e-05,
-      "loss": 0.441,
-      "mean_token_accuracy": 0.860273253917694,
-      "num_tokens": 27485385.0,
       "step": 155
     },
     {
-      "epoch": 1.8604651162790697,
-      "grad_norm": 0.5142207145690918,
-      "learning_rate": 6.498383132860849e-05,
-      "loss": 0.4278,
-      "mean_token_accuracy": 0.8661178708076477,
-      "num_tokens": 28360637.0,
       "step": 160
     },
     {
-      "epoch": 1.9186046511627906,
-      "grad_norm": 0.6509299278259277,
-      "learning_rate": 6.429052866693743e-05,
-      "loss": 0.3881,
-      "mean_token_accuracy": 0.8775515556335449,
-      "num_tokens": 29230029.0,
       "step": 165
     },
     {
-      "epoch": 1.9767441860465116,
-      "grad_norm": 0.5916144847869873,
-      "learning_rate": 6.357961898248585e-05,
-      "loss": 0.4298,
-      "mean_token_accuracy": 0.8624862432479858,
-      "num_tokens": 30082134.0,
       "step": 170
     },
     {
-      "epoch": 2.0348837209302326,
-      "grad_norm": 0.5240158438682556,
-      "learning_rate": 6.285182100181375e-05,
-      "loss": 0.4223,
-      "mean_token_accuracy": 0.8699663758277894,
-      "num_tokens": 30945309.0,
       "step": 175
     },
     {
-      "epoch": 2.0930232558139537,
-      "grad_norm": 0.66953444480896,
-      "learning_rate": 6.21078705254742e-05,
-      "loss": 0.3731,
-      "mean_token_accuracy": 0.8814450383186341,
-      "num_tokens": 31798059.0,
       "step": 180
     },
     {
-      "epoch": 2.1511627906976742,
-      "grad_norm": 0.5697144865989685,
-      "learning_rate": 6.134851968412226e-05,
-      "loss": 0.3781,
-      "mean_token_accuracy": 0.8790240526199341,
-      "num_tokens": 32700506.0,
       "step": 185
     },
     {
-      "epoch": 2.2093023255813953,
-      "grad_norm": 0.5728302001953125,
-      "learning_rate": 6.0574536178114176e-05,
-      "loss": 0.3828,
-      "mean_token_accuracy": 0.8748290300369262,
-      "num_tokens": 33560105.0,
       "step": 190
     },
     {
-      "epoch": 2.2674418604651163,
-      "grad_norm": 0.5915728807449341,
-      "learning_rate": 5.978670250136575e-05,
-      "loss": 0.3966,
-      "mean_token_accuracy": 0.8702341914176941,
-      "num_tokens": 34453105.0,
       "step": 195
     },
     {
-      "epoch": 2.3255813953488373,
-      "grad_norm": 0.6768730878829956,
-      "learning_rate": 5.898581515025438e-05,
-      "loss": 0.3926,
-      "mean_token_accuracy": 0.8739870071411133,
-      "num_tokens": 35321541.0,
       "step": 200
     },
     {
-      "epoch": 2.383720930232558,
-      "grad_norm": 0.6103410124778748,
-      "learning_rate": 5.8172683818364765e-05,
-      "loss": 0.3922,
-      "mean_token_accuracy": 0.8750384330749512,
-      "num_tokens": 36269855.0,
       "step": 205
     },
     {
-      "epoch": 2.441860465116279,
-      "grad_norm": 0.710746169090271,
-      "learning_rate": 5.734813057789223e-05,
-      "loss": 0.3572,
-      "mean_token_accuracy": 0.882230567932129,
-      "num_tokens": 37156235.0,
       "step": 210
     },
     {
-      "epoch": 2.5,
-      "grad_norm": 0.6437317132949829,
-      "learning_rate": 5.651298904853137e-05,
-      "loss": 0.369,
-      "mean_token_accuracy": 0.8787329792976379,
-      "num_tokens": 38070007.0,
       "step": 215
     },
     {
-      "epoch": 2.558139534883721,
-      "grad_norm": 0.7597388029098511,
-      "learning_rate": 5.5668103554690244e-05,
-      "loss": 0.3664,
-      "mean_token_accuracy": 0.8813632965087891,
-      "num_tokens": 38893789.0,
       "step": 220
     },
     {
-      "epoch": 2.616279069767442,
-      "grad_norm": 0.7274823784828186,
-      "learning_rate": 5.4814328271882056e-05,
-      "loss": 0.3853,
-      "mean_token_accuracy": 0.8771765947341919,
-      "num_tokens": 39778380.0,
       "step": 225
     },
     {
-      "epoch": 2.6744186046511627,
-      "grad_norm": 0.6335929036140442,
-      "learning_rate": 5.395252636315752e-05,
-      "loss": 0.3582,
-      "mean_token_accuracy": 0.8801517844200134,
-      "num_tokens": 40736100.0,
       "step": 230
     },
     {
-      "epoch": 2.7325581395348837,
-      "grad_norm": 0.6215701699256897,
-      "learning_rate": 5.3083569106450794e-05,
-      "loss": 0.3791,
-      "mean_token_accuracy": 0.8791951894760132,
-      "num_tokens": 41627731.0,
       "step": 235
     },
     {
-      "epoch": 2.7906976744186047,
-      "grad_norm": 0.6210712194442749,
-      "learning_rate": 5.220833501372133e-05,
-      "loss": 0.3515,
-      "mean_token_accuracy": 0.8856444358825684,
-      "num_tokens": 42480686.0,
       "step": 240
     },
     {
-      "epoch": 2.8488372093023253,
-      "grad_norm": 0.7439652681350708,
-      "learning_rate": 5.132770894278215e-05,
-      "loss": 0.3675,
-      "mean_token_accuracy": 0.8832207322120667,
-      "num_tokens": 43371025.0,
       "step": 245
     },
     {
-      "epoch": 2.9069767441860463,
-      "grad_norm": 0.6188499927520752,
-      "learning_rate": 5.044258120271251e-05,
-      "loss": 0.352,
-      "mean_token_accuracy": 0.8845577120780945,
-      "num_tokens": 44272951.0,
       "step": 250
     },
     {
-      "epoch": 2.9651162790697674,
-      "grad_norm": 0.6540551781654358,
-      "learning_rate": 4.955384665375935e-05,
-      "loss": 0.3452,
-      "mean_token_accuracy": 0.88858562707901,
-      "num_tokens": 45112833.0,
       "step": 255
     },
     {
-      "epoch": 3.0232558139534884,
-      "grad_norm": 0.6577905416488647,
-      "learning_rate": 4.866240380263745e-05,
-      "loss": 0.3457,
-      "mean_token_accuracy": 0.8881999373435974,
-      "num_tokens": 46027094.0,
       "step": 260
     },
     {
-      "epoch": 3.0813953488372094,
-      "grad_norm": 0.6692069172859192,
-      "learning_rate": 4.776915389414318e-05,
-      "loss": 0.3019,
-      "mean_token_accuracy": 0.8964720010757447,
-      "num_tokens": 46908302.0,
       "step": 265
     },
     {
-      "epoch": 3.13953488372093,
-      "grad_norm": 0.7803618311882019,
-      "learning_rate": 4.6874999999999994e-05,
-      "loss": 0.3246,
-      "mean_token_accuracy": 0.8924661874771118,
-      "num_tokens": 47718514.0,
       "step": 270
     },
     {
-      "epoch": 3.197674418604651,
-      "grad_norm": 0.7068886756896973,
-      "learning_rate": 4.598084610585681e-05,
-      "loss": 0.3173,
-      "mean_token_accuracy": 0.8940881609916687,
-      "num_tokens": 48605257.0,
       "step": 275
     },
     {
-      "epoch": 3.255813953488372,
-      "grad_norm": 0.7091385722160339,
-      "learning_rate": 4.5087596197362554e-05,
-      "loss": 0.3168,
-      "mean_token_accuracy": 0.8936636090278626,
-      "num_tokens": 49456564.0,
       "step": 280
     },
     {
-      "epoch": 3.313953488372093,
-      "grad_norm": 0.6630744338035583,
-      "learning_rate": 4.4196153346240655e-05,
-      "loss": 0.2912,
-      "mean_token_accuracy": 0.9037835001945496,
-      "num_tokens": 50363507.0,
       "step": 285
     },
     {
-      "epoch": 3.3720930232558137,
-      "grad_norm": 0.6711844205856323,
-      "learning_rate": 4.330741879728747e-05,
-      "loss": 0.3108,
-      "mean_token_accuracy": 0.8963549494743347,
-      "num_tokens": 51248723.0,
       "step": 290
     },
     {
-      "epoch": 3.4302325581395348,
-      "grad_norm": 0.7731291651725769,
-      "learning_rate": 4.2422291057217845e-05,
-      "loss": 0.3233,
-      "mean_token_accuracy": 0.8919758558273315,
-      "num_tokens": 52178931.0,
       "step": 295
     },
     {
-      "epoch": 3.488372093023256,
-      "grad_norm": 0.6275913715362549,
-      "learning_rate": 4.154166498627867e-05,
-      "loss": 0.3004,
-      "mean_token_accuracy": 0.9027432203292847,
-      "num_tokens": 53069706.0,
       "step": 300
     },
     {
-      "epoch": 3.546511627906977,
-      "grad_norm": 0.6897523403167725,
-      "learning_rate": 4.0666430893549215e-05,
-      "loss": 0.3173,
-      "mean_token_accuracy": 0.8981173038482666,
-      "num_tokens": 53953607.0,
       "step": 305
     },
     {
-      "epoch": 3.604651162790698,
-      "grad_norm": 0.6881428956985474,
-      "learning_rate": 3.979747363684247e-05,
-      "loss": 0.296,
-      "mean_token_accuracy": 0.9011698007583618,
-      "num_tokens": 54899379.0,
       "step": 310
     },
     {
-      "epoch": 3.6627906976744184,
-      "grad_norm": 0.7619199752807617,
-      "learning_rate": 3.893567172811793e-05,
-      "loss": 0.2697,
-      "mean_token_accuracy": 0.9105530261993409,
-      "num_tokens": 55775716.0,
       "step": 315
-    },
-    {
-      "epoch": 3.7209302325581395,
-      "grad_norm": 0.729427695274353,
-      "learning_rate": 3.808189644530976e-05,
-      "loss": 0.3051,
-      "mean_token_accuracy": 0.8989696025848388,
-      "num_tokens": 56661263.0,
-      "step": 320
-    },
-    {
-      "epoch": 3.7790697674418605,
-      "grad_norm": 0.7754212021827698,
-      "learning_rate": 3.723701095146863e-05,
-      "loss": 0.2984,
-      "mean_token_accuracy": 0.9011304140090942,
-      "num_tokens": 57550955.0,
-      "step": 325
-    },
-    {
-      "epoch": 3.8372093023255816,
-      "grad_norm": 0.7051293253898621,
-      "learning_rate": 3.640186942210777e-05,
-      "loss": 0.2973,
-      "mean_token_accuracy": 0.9021319508552551,
-      "num_tokens": 58438033.0,
-      "step": 330
-    },
-    {
-      "epoch": 3.895348837209302,
-      "grad_norm": 0.8103964328765869,
-      "learning_rate": 3.557731618163524e-05,
-      "loss": 0.2739,
-      "mean_token_accuracy": 0.9081305980682373,
-      "num_tokens": 59306335.0,
-      "step": 335
-    },
-    {
-      "epoch": 3.953488372093023,
-      "grad_norm": 0.764350175857544,
-      "learning_rate": 3.476418484974561e-05,
-      "loss": 0.3025,
-      "mean_token_accuracy": 0.9021538257598877,
-      "num_tokens": 60146879.0,
-      "step": 340
     }
   ],
   "logging_steps": 5,
-  "max_steps": 516,
   "num_input_tokens_seen": 0,
-  "num_train_epochs": 6,
   "save_steps": 500,
   "stateful_callbacks": {
     "TrainerControl": {
@@ -639,7 +594,7 @@
       "attributes": {}
     }
   },
-  "total_flos": 1.0229688130810675e+18,
   "train_batch_size": 48,
   "trial_name": null,
   "trial_params": null

   "best_global_step": null,
   "best_metric": null,
   "best_model_checkpoint": null,
+  "epoch": 3.95,
   "eval_steps": 500,
+  "global_step": 316,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
   "log_history": [
     {
+      "epoch": 0.0625,
+      "grad_norm": 2.886897563934326,
       "learning_rate": 1.3636363636363635e-05,
+      "loss": 1.3691,
+      "mean_token_accuracy": 0.6758252620697022,
+      "num_tokens": 967205.0,
       "step": 5
     },
     {
+      "epoch": 0.125,
+      "grad_norm": 0.7570586800575256,
       "learning_rate": 3.068181818181818e-05,
+      "loss": 1.269,
+      "mean_token_accuracy": 0.6787961244583129,
+      "num_tokens": 1921321.0,
       "step": 10
     },
     {
+      "epoch": 0.1875,
+      "grad_norm": 0.5072895288467407,
       "learning_rate": 4.7727272727272724e-05,
+      "loss": 1.1276,
+      "mean_token_accuracy": 0.7031140804290772,
+      "num_tokens": 2869536.0,
       "step": 15
     },
     {
+      "epoch": 0.25,
+      "grad_norm": 0.43829378485679626,
       "learning_rate": 6.477272727272727e-05,
+      "loss": 1.0321,
+      "mean_token_accuracy": 0.7193317890167237,
+      "num_tokens": 3858500.0,
       "step": 20
     },
     {
+      "epoch": 0.3125,
+      "grad_norm": 0.4101383090019226,
+      "learning_rate": 7.499374865230175e-05,
+      "loss": 0.9679,
+      "mean_token_accuracy": 0.734985601902008,
+      "num_tokens": 4789698.0,
       "step": 25
     },
     {
+      "epoch": 0.375,
+      "grad_norm": 0.4972609281539917,
+      "learning_rate": 7.492345290159602e-05,
+      "loss": 0.8265,
+      "mean_token_accuracy": 0.7632619500160217,
+      "num_tokens": 5778900.0,
       "step": 30
     },
     {
+      "epoch": 0.4375,
+      "grad_norm": 0.3812519609928131,
+      "learning_rate": 7.477524313699733e-05,
+      "loss": 0.777,
+      "mean_token_accuracy": 0.780980110168457,
+      "num_tokens": 6747243.0,
       "step": 35
     },
     {
+      "epoch": 0.5,
+      "grad_norm": 0.41406524181365967,
+      "learning_rate": 7.454953106098704e-05,
+      "loss": 0.745,
+      "mean_token_accuracy": 0.7827757239341736,
+      "num_tokens": 7718219.0,
       "step": 40
     },
     {
+      "epoch": 0.5625,
+      "grad_norm": 0.4349099099636078,
+      "learning_rate": 7.42469436647895e-05,
+      "loss": 0.7068,
+      "mean_token_accuracy": 0.7929372310638427,
+      "num_tokens": 8655461.0,
       "step": 45
     },
     {
+      "epoch": 0.625,
+      "grad_norm": 0.4021725356578827,
+      "learning_rate": 7.386832148669304e-05,
+      "loss": 0.7122,
+      "mean_token_accuracy": 0.7939839363098145,
+      "num_tokens": 9604900.0,
       "step": 50
     },
     {
+      "epoch": 0.6875,
+      "grad_norm": 0.4050525426864624,
+      "learning_rate": 7.341471627717199e-05,
+      "loss": 0.6396,
+      "mean_token_accuracy": 0.813041603565216,
+      "num_tokens": 10597014.0,
       "step": 55
     },
     {
+      "epoch": 0.75,
+      "grad_norm": 0.4006189703941345,
+      "learning_rate": 7.288738807729602e-05,
+      "loss": 0.6521,
+      "mean_token_accuracy": 0.8042582392692565,
+      "num_tokens": 11601956.0,
       "step": 60
     },
     {
+      "epoch": 0.8125,
+      "grad_norm": 0.41143348813056946,
+      "learning_rate": 7.228780171854215e-05,
+      "loss": 0.638,
+      "mean_token_accuracy": 0.8092883229255676,
+      "num_tokens": 12547547.0,
       "step": 65
     },
     {
+      "epoch": 0.875,
+      "grad_norm": 0.534580409526825,
+      "learning_rate": 7.161762275373244e-05,
+      "loss": 0.5964,
+      "mean_token_accuracy": 0.8182763457298279,
+      "num_tokens": 13518585.0,
       "step": 70
     },
     {
+      "epoch": 0.9375,
+      "grad_norm": 0.5157734751701355,
+      "learning_rate": 7.087871283040086e-05,
+      "loss": 0.6566,
+      "mean_token_accuracy": 0.7993115663528443,
+      "num_tokens": 14487359.0,
       "step": 75
     },
     {
+      "epoch": 1.0,
+      "grad_norm": 0.47575053572654724,
+      "learning_rate": 7.007312451944087e-05,
+      "loss": 0.5802,
+      "mean_token_accuracy": 0.8218901753425598,
+      "num_tokens": 15456737.0,
       "step": 80
     },
     {
+      "epoch": 1.0625,
+      "grad_norm": 0.5268984436988831,
+      "learning_rate": 6.92030956133992e-05,
+      "loss": 0.5475,
+      "mean_token_accuracy": 0.8278343200683593,
+      "num_tokens": 16398357.0,
       "step": 85
     },
     {
+      "epoch": 1.125,
+      "grad_norm": 0.4960674047470093,
+      "learning_rate": 6.827104291025441e-05,
+      "loss": 0.546,
+      "mean_token_accuracy": 0.828590726852417,
+      "num_tokens": 17358500.0,
       "step": 90
     },
     {
+      "epoch": 1.1875,
+      "grad_norm": 0.5320209860801697,
+      "learning_rate": 6.727955549994725e-05,
+      "loss": 0.5207,
+      "mean_token_accuracy": 0.8370408773422241,
+      "num_tokens": 18342498.0,
       "step": 95
     },
     {
+      "epoch": 1.25,
+      "grad_norm": 0.579660177230835,
+      "learning_rate": 6.623138757231244e-05,
+      "loss": 0.5369,
+      "mean_token_accuracy": 0.835267984867096,
+      "num_tokens": 19327440.0,
       "step": 100
     },
     {
+      "epoch": 1.3125,
+      "grad_norm": 0.551972508430481,
+      "learning_rate": 6.512945076638986e-05,
+      "loss": 0.5268,
+      "mean_token_accuracy": 0.8328914523124695,
+      "num_tokens": 20296574.0,
       "step": 105
     },
     {
+      "epoch": 1.375,
+      "grad_norm": 0.47652754187583923,
+      "learning_rate": 6.397680608236752e-05,
+      "loss": 0.5298,
+      "mean_token_accuracy": 0.8306650876998901,
+      "num_tokens": 21246957.0,
       "step": 110
     },
     {
+      "epoch": 1.4375,
+      "grad_norm": 0.5899170637130737,
+      "learning_rate": 6.277665537862383e-05,
+      "loss": 0.5315,
+      "mean_token_accuracy": 0.8339821338653565,
+      "num_tokens": 22189101.0,
       "step": 115
     },
     {
+      "epoch": 1.5,
+      "grad_norm": 0.5031387805938721,
+      "learning_rate": 6.153233247748881e-05,
+      "loss": 0.4929,
+      "mean_token_accuracy": 0.8448952674865723,
+      "num_tokens": 23178128.0,
       "step": 120
     },
     {
+      "epoch": 1.5625,
+      "grad_norm": 0.5996953248977661,
+      "learning_rate": 6.0247293904430945e-05,
+      "loss": 0.4807,
+      "mean_token_accuracy": 0.8454113602638245,
+      "num_tokens": 24150209.0,
       "step": 125
     },
     {
+      "epoch": 1.625,
+      "grad_norm": 0.5869215726852417,
+      "learning_rate": 5.892510928639492e-05,
+      "loss": 0.4759,
+      "mean_token_accuracy": 0.8487537622451782,
+      "num_tokens": 25114253.0,
       "step": 130
     },
     {
+      "epoch": 1.6875,
+      "grad_norm": 0.6149603128433228,
+      "learning_rate": 5.756945143596188e-05,
+      "loss": 0.4714,
+      "mean_token_accuracy": 0.8475272536277771,
+      "num_tokens": 26051613.0,
       "step": 135
     },
     {
+      "epoch": 1.75,
+      "grad_norm": 0.6760532259941101,
+      "learning_rate": 5.6184086148876894e-05,
+      "loss": 0.5068,
+      "mean_token_accuracy": 0.8413119912147522,
+      "num_tokens": 26990255.0,
       "step": 140
     },
     {
+      "epoch": 1.8125,
+      "grad_norm": 0.6503332853317261,
+      "learning_rate": 5.477286174328442e-05,
+      "loss": 0.4402,
+      "mean_token_accuracy": 0.8568204998970032,
+      "num_tokens": 27999730.0,
       "step": 145
     },
     {
+      "epoch": 1.875,
+      "grad_norm": 0.611965000629425,
+      "learning_rate": 5.333969836972997e-05,
+      "loss": 0.4814,
+      "mean_token_accuracy": 0.8444833040237427,
+      "num_tokens": 28967267.0,
       "step": 150
     },
     {
+      "epoch": 1.9375,
+      "grad_norm": 0.6395599842071533,
+      "learning_rate": 5.188857712162322e-05,
+      "loss": 0.4613,
+      "mean_token_accuracy": 0.8521966099739074,
+      "num_tokens": 29916924.0,
       "step": 155
     },
     {
+      "epoch": 2.0,
+      "grad_norm": 0.670397937297821,
+      "learning_rate": 5.042352897641175e-05,
+      "loss": 0.4742,
+      "mean_token_accuracy": 0.8465383648872375,
+      "num_tokens": 30895018.0,
       "step": 160
     },
     {
+      "epoch": 2.0625,
+      "grad_norm": 0.6061509847640991,
+      "learning_rate": 4.894862359818504e-05,
+      "loss": 0.4264,
+      "mean_token_accuracy": 0.8613315701484681,
+      "num_tokens": 31832051.0,
       "step": 165
     },
     {
+      "epoch": 2.125,
+      "grad_norm": 0.6346914768218994,
+      "learning_rate": 4.746795803281343e-05,
+      "loss": 0.4153,
+      "mean_token_accuracy": 0.8654327511787414,
+      "num_tokens": 32782084.0,
       "step": 170
     },
     {
+      "epoch": 2.1875,
+      "grad_norm": 0.6239711046218872,
+      "learning_rate": 4.59856453270249e-05,
+      "loss": 0.4147,
+      "mean_token_accuracy": 0.8632163763046264,
+      "num_tokens": 33740572.0,
       "step": 175
     },
     {
+      "epoch": 2.25,
+      "grad_norm": 0.5688380002975464,
+      "learning_rate": 4.450580310303404e-05,
+      "loss": 0.4035,
+      "mean_token_accuracy": 0.864958119392395,
+      "num_tokens": 34760832.0,
       "step": 180
     },
     {
+      "epoch": 2.3125,
+      "grad_norm": 0.6970889568328857,
+      "learning_rate": 4.303254212046112e-05,
+      "loss": 0.4278,
+      "mean_token_accuracy": 0.8623119592666626,
+      "num_tokens": 35675387.0,
       "step": 185
     },
     {
+      "epoch": 2.375,
+      "grad_norm": 0.6932367086410522,
+      "learning_rate": 4.156995485731436e-05,
+      "loss": 0.4167,
+      "mean_token_accuracy": 0.8627482533454895,
+      "num_tokens": 36654691.0,
       "step": 190
     },
     {
+      "epoch": 2.4375,
+      "grad_norm": 0.74919193983078,
+      "learning_rate": 4.0122104141755454e-05,
+      "loss": 0.4041,
+      "mean_token_accuracy": 0.8695870995521545,
+      "num_tokens": 37623031.0,
       "step": 195
     },
     {
+      "epoch": 2.5,
+      "grad_norm": 0.6283291578292847,
+      "learning_rate": 3.869301186622754e-05,
+      "loss": 0.3863,
+      "mean_token_accuracy": 0.8726918458938598,
+      "num_tokens": 38620801.0,
       "step": 200
     },
     {
+      "epoch": 2.5625,
+      "grad_norm": 0.5696513652801514,
+      "learning_rate": 3.728664781529596e-05,
+      "loss": 0.3979,
+      "mean_token_accuracy": 0.8664975047111512,
+      "num_tokens": 39619709.0,
       "step": 205
     },
     {
+      "epoch": 2.625,
+      "grad_norm": 0.804267168045044,
+      "learning_rate": 3.590691863823604e-05,
+      "loss": 0.4252,
+      "mean_token_accuracy": 0.8591230988502503,
+      "num_tokens": 40603368.0,
       "step": 210
     },
     {
+      "epoch": 2.6875,
+      "grad_norm": 0.8184316158294678,
+      "learning_rate": 3.4557656997000574e-05,
+      "loss": 0.4003,
+      "mean_token_accuracy": 0.8659304857254029,
+      "num_tokens": 41548896.0,
       "step": 215
     },
     {
+      "epoch": 2.75,
+      "grad_norm": 0.8127630352973938,
+      "learning_rate": 3.324261091971178e-05,
+      "loss": 0.3871,
+      "mean_token_accuracy": 0.8709235548973083,
+      "num_tokens": 42524953.0,
       "step": 220
     },
     {
+      "epoch": 2.8125,
+      "grad_norm": 0.7622901797294617,
+      "learning_rate": 3.1965433389252365e-05,
+      "loss": 0.3845,
+      "mean_token_accuracy": 0.8702649950981141,
+      "num_tokens": 43484492.0,
       "step": 225
     },
     {
+      "epoch": 2.875,
+      "grad_norm": 0.6691771149635315,
+      "learning_rate": 3.0729672195876615e-05,
+      "loss": 0.3907,
+      "mean_token_accuracy": 0.8697134971618652,
+      "num_tokens": 44459752.0,
       "step": 230
     },
     {
+      "epoch": 2.9375,
+      "grad_norm": 0.8623871207237244,
+      "learning_rate": 2.953876008202943e-05,
+      "loss": 0.4045,
+      "mean_token_accuracy": 0.8645989656448364,
+      "num_tokens": 45374136.0,
       "step": 235
     },
     {
+      "epoch": 3.0,
+      "grad_norm": 0.6397231817245483,
+      "learning_rate": 2.8396005206749356e-05,
+      "loss": 0.3915,
+      "mean_token_accuracy": 0.8694759726524353,
+      "num_tokens": 46347054.0,
       "step": 240
     },
     {
+      "epoch": 3.0625,
+      "grad_norm": 0.6764381527900696,
+      "learning_rate": 2.7304581956143917e-05,
+      "loss": 0.3568,
+      "mean_token_accuracy": 0.8802210927009583,
+      "num_tokens": 47320044.0,
       "step": 245
     },
     {
+      "epoch": 3.125,
+      "grad_norm": 0.7345840334892273,
+      "learning_rate": 2.626752212546434e-05,
+      "loss": 0.3549,
+      "mean_token_accuracy": 0.878583550453186,
+      "num_tokens": 48309769.0,
       "step": 250
     },
     {
+      "epoch": 3.1875,
+      "grad_norm": 0.8476727604866028,
+      "learning_rate": 2.528770649727422e-05,
+      "loss": 0.3341,
+      "mean_token_accuracy": 0.8889845728874206,
+      "num_tokens": 49276272.0,
       "step": 255
     },
     {
+      "epoch": 3.25,
+      "grad_norm": 0.825459361076355,
+      "learning_rate": 2.4367856839106895e-05,
+      "loss": 0.3821,
+      "mean_token_accuracy": 0.8741106748580932,
+      "num_tokens": 50223204.0,
       "step": 260
     },
     {
+      "epoch": 3.3125,
+      "grad_norm": 0.7627800703048706,
+      "learning_rate": 2.3510528342840405e-05,
+      "loss": 0.3394,
+      "mean_token_accuracy": 0.8844169974327087,
+      "num_tokens": 51217236.0,
       "step": 265
     },
     {
+      "epoch": 3.375,
+      "grad_norm": 0.7601186037063599,
+      "learning_rate": 2.2718102526792474e-05,
+      "loss": 0.334,
+      "mean_token_accuracy": 0.8848990678787232,
+      "num_tokens": 52214250.0,
       "step": 270
     },
     {
+      "epoch": 3.4375,
+      "grad_norm": 1.0226414203643799,
+      "learning_rate": 2.1992780620252184e-05,
+      "loss": 0.3371,
+      "mean_token_accuracy": 0.8848458886146545,
+      "num_tokens": 53147457.0,
       "step": 275
     },
     {
+      "epoch": 3.5,
+      "grad_norm": 0.6799741387367249,
+      "learning_rate": 2.1336577448825122e-05,
+      "loss": 0.3391,
+      "mean_token_accuracy": 0.884827721118927,
+      "num_tokens": 54144048.0,
       "step": 280
     },
     {
+      "epoch": 3.5625,
+      "grad_norm": 0.7272220849990845,
+      "learning_rate": 2.075131583757742e-05,
+      "loss": 0.3392,
+      "mean_token_accuracy": 0.8837083220481873,
+      "num_tokens": 55118118.0,
       "step": 285
     },
     {
+      "epoch": 3.625,
+      "grad_norm": 0.7519774436950684,
+      "learning_rate": 2.0238621547525902e-05,
+      "loss": 0.3381,
+      "mean_token_accuracy": 0.8849397540092468,
+      "num_tokens": 56034052.0,
       "step": 290
     },
     {
+      "epoch": 3.6875,
+      "grad_norm": 0.917468249797821,
+      "learning_rate": 1.979991875953994e-05,
+      "loss": 0.3109,
+      "mean_token_accuracy": 0.8913192272186279,
+      "num_tokens": 56994081.0,
       "step": 295
     },
     {
+      "epoch": 3.75,
+      "grad_norm": 0.7441599369049072,
+      "learning_rate": 1.943642611819996e-05,
+      "loss": 0.3364,
+      "mean_token_accuracy": 0.8865782380104065,
+      "num_tokens": 57997199.0,
       "step": 300
     },
     {
+      "epoch": 3.8125,
+      "grad_norm": 0.9184499382972717,
+      "learning_rate": 1.9149153346602176e-05,
+      "loss": 0.3488,
+      "mean_token_accuracy": 0.8831375479698181,
+      "num_tokens": 58922928.0,
       "step": 305
     },
     {
+      "epoch": 3.875,
+      "grad_norm": 0.8689393997192383,
+      "learning_rate": 1.893889844151295e-05,
+      "loss": 0.3406,
+      "mean_token_accuracy": 0.884752631187439,
+      "num_tokens": 59846729.0,
       "step": 310
     },
     {
+      "epoch": 3.9375,
+      "grad_norm": 0.822659432888031,
+      "learning_rate": 1.8806245456664362e-05,
+      "loss": 0.336,
+      "mean_token_accuracy": 0.8861027598381043,
+      "num_tokens": 60832044.0,
       "step": 315
     }
   ],
   "logging_steps": 5,
+  "max_steps": 320,
   "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
   "save_steps": 500,
   "stateful_callbacks": {
     "TrainerControl": {
       "attributes": {}
     }
   },
+  "total_flos": 1.0379419406848492e+18,
   "train_batch_size": 48,
   "trial_name": null,
   "trial_params": null

training_args.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:11b2dc08c61f0b0bd6c9cf3944edb9af5b3264a14ba67c0aff78512d95aa186b
 size 6417

 version https://git-lfs.github.com/spec/v1
+oid sha256:c8fb0fe44069b4bae4e0c71790211b8428792f864c7811e300552e9a7dba13cc
 size 6417