File size: 5,010 Bytes
8cb2159 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | {
"cases": [
{
"expected": {
"greedy_completion_token_ids_sha256": "92a82b78a512b7938e06f2f596cd1777db4c69413a63ca229d342d81a63986ac",
"input_ids_sha256": "a63e87353ed75a1600338816a9b1574bdeeb710ecf9e2153e1c605af31e06683",
"rendered_prompt_sha256": "bae4f1e9c5678fb4d36621a7bda568e52ce7ad5a0b723d1f580f236ec75adda4"
},
"id": "short_arithmetic",
"messages": [
{
"content": "Compute 37 times 41 and give one short arithmetic check.",
"role": "user"
}
]
},
{
"expected": {
"greedy_completion_token_ids_sha256": "b82c420abae15addad25edbeab119a90cd5c722d55a77e39e40a94b900108c77",
"input_ids_sha256": "b5494568bee147af3c7305f9082940d065bb16211e424e80416e949147552880",
"rendered_prompt_sha256": "c213d17e6fac34c0e6501aee9caa6808d6be6d8926b546cc473af8c5bb061094"
},
"id": "algebra",
"messages": [
{
"content": "Solve x squared minus 5x plus 6 equals zero over the real numbers.",
"role": "user"
}
]
},
{
"expected": {
"greedy_completion_token_ids_sha256": "6f3555f8199e56e939997e2b0a01a8cf6eddf6885dc27f270ef81628faa2b93a",
"input_ids_sha256": "62dfce834d0d73e36f64dfff89588ab3237383d0398d6294e43fb0b878d92bbc",
"rendered_prompt_sha256": "37e32f79791b135f8229fe6436f4bca8dbada804a0925d7e6d113a677b76ca29"
},
"id": "proof_style",
"messages": [
{
"content": "Prove that the sum of the first n odd integers is n squared.",
"role": "user"
}
]
},
{
"expected": {
"greedy_completion_token_ids_sha256": "caefbfb4b898919d0469db72f03606a94aaa4bcfd61c0646e80d8e274c7514ca",
"input_ids_sha256": "3408d510f35797d3f82927942c957818276167e718ab30a67c7c8252dd5ceea2",
"rendered_prompt_sha256": "2390c8f9293a842bc31edcf0cb8ae886ab6b97ca22bb129d882d29ecf6f83cd6"
},
"id": "long_context_formatting",
"messages": [
{
"content": "Read the following synthetic ledger. It is context for a formatting test, not a benchmark. Entry 01 defines a_1 = 3. Entry 02 says each a_{k+1} is a_k + 2. Entry 03 records b_1 = 1. Entry 04 says each b_{k+1} is b_k + 3. Entry 05 defines c_k = a_k + b_k. Entry 06 states that all indices are positive integers. Entry 07 says no term is omitted. Entry 08 says arithmetic is exact. Entry 09 defines d_k = a_k - b_k. Entry 10 asks readers to preserve subscripts. Entry 11 records a_2 = 5. Entry 12 records b_2 = 4. Entry 13 records c_2 = 9. Entry 14 records d_2 = 1. Entry 15 records a_3 = 7. Entry 16 records b_3 = 7. Entry 17 records c_3 = 14. Entry 18 records d_3 = 0. Entry 19 records a_4 = 9. Entry 20 records b_4 = 10. Entry 21 records c_4 = 19. Entry 22 records d_4 = -1. Entry 23 says a_k is an arithmetic progression. Entry 24 says b_k is another arithmetic progression. Entry 25 says c_k combines the two progressions termwise. Entry 26 says d_k compares them termwise. Entry 27 requires a symbolic answer before a numerical example. Entry 28 requires the numerical example to use k = 25. Entry 29 says the final response should distinguish derivation from substitution. Entry 30 says the final response should not invent any additional sequence. Entry 31 repeats that a_1 = 3 and the common difference of a is 2. Entry 32 repeats that b_1 = 1 and the common difference of b is 3. Entry 33 notes that a_k therefore grows more slowly than b_k. Entry 34 notes that d_k eventually becomes negative. Entry 35 says c_k remains positive. Entry 36 says every formula should be valid for k at least 1. Entry 37 asks that parentheses be used when substituting k = 25. Entry 38 asks that the conclusion fit in three sentences. Entry 39 says the ledger contains deliberate redundancy. Entry 40 says redundant entries do not change the definitions. Entry 41 records that the requested quantity is c_k, not d_k. Entry 42 warns not to sum the first k terms. Entry 43 warns not to multiply a_k and b_k. Entry 44 says only termwise addition is relevant. Entry 45 notes that a_k = 3 + 2(k - 1). Entry 46 notes that b_k = 1 + 3(k - 1). Entry 47 permits simplifying each expression. Entry 48 permits simplifying c_k after addition. Entry 49 says the numerical answer must follow from the symbolic one. Entry 50 closes the synthetic ledger.",
"role": "user"
},
{
"content": "I have read the ledger and will use only its definitions.",
"role": "assistant"
},
{
"content": "Derive a closed form for c_k and then evaluate c_25, following the requested format.",
"role": "user"
}
]
}
],
"filled_by": "scripts/hf_release/validate_equivalence.py",
"generation": {
"do_sample": false,
"max_new_tokens": 16
},
"purpose": "multi-case merged-versus-adapter and clean-download validation",
"schema_version": "ulam-public-inference-fixture-v2"
}
|