Hugging Face's logo

josephmayo
/
ZAYA1-8B-Coder

Text Generation
Transformers
Safetensors
zaya
lora
coder
merged
conversational
Model card Files
xet
 Community
josephmayo commited on
Commit
9cf1a6e
·
verified ·
1 Parent(s): 39d8be5

Add local evidence artifacts and model-card evidence links

Browse files
Files changed (3) hide show
  1. README.md +91 -79
  2. evidence/README.md +5 -0
  3. evidence/zaya_qlora_eval_result_zaya1-8b-coding-qlora-eval_release_summary.json +103 -0
README.md CHANGED
@@ -1,79 +1,91 @@
1
- ---
2
- base_model: Zyphra/ZAYA1-8B
3
- library_name: transformers
4
- tags:
5
- - zaya
6
- - lora
7
- - coder
8
- - merged
9
- ---
10
-
11
- # ZAYA1-8B Coder
12
-
13
- Merged Coder model from `Zyphra/ZAYA1-8B` and `josephmayo/ZAYA1-8B-Coder-LoRA`. This repo contains the adapter merged into the base weights as normal safetensors shards.
14
-
15
- ## Evaluation Gate
16
-
17
- The adapter was evaluated against the base model on 50 Python code-generation prompts with a 0-10 heuristic score:
18
-
19
- - Base average: 2.36 / 10
20
- - LoRA average: 4.76 / 10
21
- - Absolute score delta: +2.40 / 10
22
- - Full-scale lift: 24.00%
23
- - Relative lift over base average: 101.69%
24
- - Improved prompts: 39 / 50
25
- - Merge threshold: 20.00%
26
- - Merge decision: true
27
-
28
- Full-scale lift is the required notebook metric:
29
-
30
- ```text
31
- ((lora_avg - base_avg) / 10) * 100
32
- ((4.76 - 2.36) / 10) * 100 = 24.00%
33
- ```
34
-
35
- ## Scoring Heuristic
36
-
37
- Each response was scored out of 10:
38
-
39
- - `def` present: 2 points
40
- - `class` present: 1 point
41
- - `return` present: 1 point
42
- - `import` or `from` present: 1 point
43
- - fenced code block present: 1 point
44
- - output length greater than 100 characters: 1 point
45
- - Python AST parse validity: 3 points
46
-
47
- ## Architecture Notes
48
-
49
- ZAYA uses a custom `model_type = zaya`; it is not weight-compatible with `LlamaForCausalLM` despite similar naming in some configs. During evaluation and merge, the real ZAYA architecture was loaded using Zyphra's Transformers implementation:
50
-
51
- ```bash
52
- pip install git+https://github.com/Zyphra/transformers.git@zaya1
53
- ```
54
-
55
- The LoRA adapter contains 160 tensors targeting:
56
-
57
- - `self_attn.o_proj`
58
- - `zaya_block.router.down_proj`
59
-
60
- The merge was performed after the evaluation gate passed, then the merged model was saved to safetensors shards with tokenizer and generation config.
61
-
62
- Evaluation artifacts are included under `eval/`:
63
-
64
- - `eval/eval_summary.json`
65
- - `eval/score_table.csv`
66
- - `eval/base_outputs.jsonl`
67
- - `eval/lora_outputs.jsonl`
68
-
69
- ## Included Files
70
-
71
- - `model-00001-of-00005.safetensors` through `model-00005-of-00005.safetensors`
72
- - `model.safetensors.index.json`
73
- - `config.json`
74
- - `generation_config.json`
75
- - tokenizer files
76
- - `zaya_patched_config.json`
77
- - evaluation outputs under `eval/`
78
-
79
- The GGUF quantized release is available at `josephmayo/ZAYA1-8B-Coder-GGUF`.
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Zyphra/ZAYA1-8B
3
+ library_name: transformers
4
+ tags:
5
+ - zaya
6
+ - lora
7
+ - coder
8
+ - merged
9
+ ---
10
+
11
+ # ZAYA1-8B Coder
12
+
13
+ Merged Coder model from `Zyphra/ZAYA1-8B` and `josephmayo/ZAYA1-8B-Coder-LoRA`. This repo contains the adapter merged into the base weights as normal safetensors shards.
14
+
15
+ ## Evaluation Gate
16
+
17
+ The adapter was evaluated against the base model on 50 Python code-generation prompts with a 0-10 heuristic score:
18
+
19
+ - Base average: 2.36 / 10
20
+ - LoRA average: 4.76 / 10
21
+ - Absolute score delta: +2.40 / 10
22
+ - Full-scale lift: 24.00%
23
+ - Relative lift over base average: 101.69%
24
+ - Improved prompts: 39 / 50
25
+ - Merge threshold: 20.00%
26
+ - Merge decision: true
27
+
28
+ Full-scale lift is the required notebook metric:
29
+
30
+ ```text
31
+ ((lora_avg - base_avg) / 10) * 100
32
+ ((4.76 - 2.36) / 10) * 100 = 24.00%
33
+ ```
34
+
35
+ ## Scoring Heuristic
36
+
37
+ Each response was scored out of 10:
38
+
39
+ - `def` present: 2 points
40
+ - `class` present: 1 point
41
+ - `return` present: 1 point
42
+ - `import` or `from` present: 1 point
43
+ - fenced code block present: 1 point
44
+ - output length greater than 100 characters: 1 point
45
+ - Python AST parse validity: 3 points
46
+
47
+ ## Architecture Notes
48
+
49
+ ZAYA uses a custom `model_type = zaya`; it is not weight-compatible with `LlamaForCausalLM` despite similar naming in some configs. During evaluation and merge, the real ZAYA architecture was loaded using Zyphra's Transformers implementation:
50
+
51
+ ```bash
52
+ pip install git+https://github.com/Zyphra/transformers.git@zaya1
53
+ ```
54
+
55
+ The LoRA adapter contains 160 tensors targeting:
56
+
57
+ - `self_attn.o_proj`
58
+ - `zaya_block.router.down_proj`
59
+
60
+ The merge was performed after the evaluation gate passed, then the merged model was saved to safetensors shards with tokenizer and generation config.
61
+
62
+ Evaluation artifacts are included under `eval/`:
63
+
64
+ - `eval/eval_summary.json`
65
+ - `eval/score_table.csv`
66
+ - `eval/base_outputs.jsonl`
67
+ - `eval/lora_outputs.jsonl`
68
+
69
+ ## Included Files
70
+
71
+ - `model-00001-of-00005.safetensors` through `model-00005-of-00005.safetensors`
72
+ - `model.safetensors.index.json`
73
+ - `config.json`
74
+ - `generation_config.json`
75
+ - tokenizer files
76
+ - `zaya_patched_config.json`
77
+ - evaluation outputs under `eval/`
78
+
79
+ The GGUF quantized release is available at `josephmayo/ZAYA1-8B-Coder-GGUF`.
80
+
81
+ <!-- evidence-files:start -->
82
+
83
+ ## Evidence files
84
+
85
+ Run evidence for this release is stored in the repository under `evidence/`:
86
+
87
+ - [`evidence/zaya_qlora_eval_result_zaya1-8b-coding-qlora-eval_release_summary.json`](./evidence/zaya_qlora_eval_result_zaya1-8b-coding-qlora-eval_release_summary.json)
88
+
89
+ These files are compact local/Kaggle run artifacts used to document training, evaluation, merge, or quantization evidence for this model family.
90
+
91
+ <!-- evidence-files:end -->
evidence/README.md ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ # Evidence files
2
+
3
+ This folder contains local training/evaluation/merge/quantization evidence artifacts copied from the original run outputs.
4
+
5
+ - `evidence/zaya_qlora_eval_result_zaya1-8b-coding-qlora-eval_release_summary.json`
evidence/zaya_qlora_eval_result_zaya1-8b-coding-qlora-eval_release_summary.json ADDED
@@ -0,0 +1,103 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run_slug": "zaya1-8b-coding-qlora-eval",
3
+ "model_id": "Zyphra/ZAYA1-8B",
4
+ "stage": "done",
5
+ "errors": [],
6
+ "cuda": true,
7
+ "devices": [
8
+ "Tesla T4",
9
+ "Tesla T4"
10
+ ],
11
+ "num_prompts": 10,
12
+ "base_done": true,
13
+ "lora_done": true,
14
+ "comparisons": [
15
+ {
16
+ "prompt": "Write a Python function that sorts a list of integers using quicksort.",
17
+ "base_output": "</think>\n</think>\n<\n</think>\n\nOkay\n\nuser\n</think>\n\n...\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n(\n\n",
18
+ "lora_output": "</think>\n\n<think>\n</think>\n\n```python\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing import Dict\nfrom typing",
19
+ "base_score": 0,
20
+ "lora_score": 2,
21
+ "improved": true
22
+ },
23
+ {
24
+ "prompt": "Implement a binary search algorithm in Python.",
25
+ "base_output": "</think>\n</think>\n\n\n\n<\n</think>\n\n\n\n...\n\nOkay\n\n\n</think>\n\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n",
26
+ "lora_output": "</think>\n\n<think>\n</think>\n\n```python\ndef binary_search(arr, target):\n low = LowLow\n left = LowLow\n while True:\n mid = (left + right) // Low\n if target == arr[mid]:\n return mid\n elif target < arr[mid]:\n right = mid\n else:\n left = mid\n if left >= right:\n return None\n return mid\n```",
27
+ "base_score": 0,
28
+ "lora_score": 5,
29
+ "improved": true
30
+ },
31
+ {
32
+ "prompt": "Write a Python class for a simple stack with push, pop, and peek methods.",
33
+ "base_output": "</think>\n None\n</think>\n\n</think>\n</think>\n</think>\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n</think>\n\n</think>\n</think>\n\n</think>\n\n</think>\n</think>\n\n</think>\n</think>\n</think>\n\n</think>\n</think>\n</think>\n\n</think>\n</think>\n</think>\n\n</think>\n</think>\n</think>\n</think>\n</think>\n</think>\n\n</think>\n</think>\n</think>\n\n</think>\n\n</think>\n</think>\n</think>\n\n</think>\n\n</think>\n</think>\n</think>\n\n</think>\n</think>\n</think>\n</think>\n\n</think>\n</think>\n</think>\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n</think>\n</think>\n</think>\n</think>\n</think>\n</think>\n\n</think>\n\n</think>\n\n</think>\n</think>\n</think>",
34
+ "lora_output": "</think>\n\n```python\nclass Stack:\n def __init__(self):\n self.stack = []\n\n def push(self,value):\n self.stack.append(value)\n\n def pop(self):\n self.stack.pop()\n\n def peek(self):\n self.stack.pop()\n\n def peek(self):\n self.stack.pop()\n\n return self.stack[-1] if return None\n```",
35
+ "base_score": 0,
36
+ "lora_score": 5,
37
+ "improved": true
38
+ },
39
+ {
40
+ "prompt": "Generate a Python function to check if a string is a palindrome.",
41
+ "base_output": "</think>\nOkay\n<\n\n```\n</think>\n\n\n\n\n\n\n```\n</think>\n\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n",
42
+ "lora_output": "</think>\n\n```python\ndef is_string(value):\n return isinstance(value) and\n```",
43
+ "base_score": 2,
44
+ "lora_score": 5,
45
+ "improved": true
46
+ },
47
+ {
48
+ "prompt": "Write a Python function to compute the factorial of n using recursion.",
49
+ "base_output": "</think>\nOkay\n \n</think>\n\n(\nOkay\n free\ncpp\ncomputer\ndebug\nassistant\n</think>\n\n...\n\n[\n</think>\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free\n...\n\n...\n\n[\n free",
50
+ "lora_output": "</think>\n\n```python\ndef compute_n(n):\n if n == \n return n\n return n + \n```",
51
+ "base_score": 0,
52
+ "lora_score": 5,
53
+ "improved": true
54
+ },
55
+ {
56
+ "prompt": "Implement a Python function to find the longest common subsequence of two strings.",
57
+ "base_output": "</think>\n</think>\n\n\nOkay\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n</\n</think>\n\n</\n</\n</think>\n\n</\n</\n</think>\n\n</\n</\n</think>\n\n</\n</\n</think>\n\n</\n</\n</think>\n\n</\n</\n</think>\n\n</\n</\n</think>\n\n</\n</\n</think>\n\n</\n</\n</think>\n\n</\n</\n</\n</think>\n\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n</\n",
58
+ "lora_output": "</think>\n\n```python\ndef longest_common_substring(s1, s):\n if len(s) < len(s):\n return \"\"\n if len(s) == (s):\n return s\n if len(s) == (s):\n return s\n if len(s) == (s):\n return s\n if len(s) == (s):\n return s\n if len(s) == (s):\n return s\n if len(s) == (s):\n return s\n if len(s) == (s):\n return s\n if len(s) == (s):\n return s\n if len(s) == (s):\n return s\n if len(s) == (s):\n return s\n if len(s) == (s):\n return s\n if len(s) == (s):\n return s\n if len(s) == (s):\n return s\n if len(s) == (s):\n return",
59
+ "base_score": 0,
60
+ "lora_score": 5,
61
+ "improved": true
62
+ },
63
+ {
64
+ "prompt": "Write a Python function to merge two sorted lists into one sorted list.",
65
+ "base_output": "Okay\n</think>\n\n</think>\n\n</think>\n\n</think>\n\n<\n</think>\n\n</think>\n\n</\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n",
66
+ "lora_output": "</think>\n\n```python\ndef merge_sorted(a, b):\n merged = []\n i = \n j = L\n while i < len(a) and b == len(b):\n if a[i] < b[j]:\n append.append(a[i])\n i += \n elif b[j]:\n if a[i] < b[j]:\n items.append(a[i])\n i += \n items.append(b[j])\n j += L\n else items.append(b[j])\n j += L\n items.append(a[i])\n i += L\n items.append(b[j])\n j += L\n while i < len(a):\n items.append(a[i])\n i += L\n while j < len(b):\n items.append(b[j])\n j += L\n return items\n```",
67
+ "base_score": 0,
68
+ "lora_score": 5,
69
+ "improved": true
70
+ },
71
+ {
72
+ "prompt": "Create a Python function to check if a number is prime.",
73
+ "base_output": "Okay\nimport\npython\n```\n\n\n</think>\n\npython\n\n\n```\n\n</think>\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n",
74
+ "lora_output": "</think>\n\n```python\ndef is_prime(n):\n if n < (0):\n return False\n while n % (2) == (0):\n n = n // (2)\n return n == (1)\n```",
75
+ "base_score": 3,
76
+ "lora_score": 5,
77
+ "improved": true
78
+ },
79
+ {
80
+ "prompt": "Write a Python function to flatten a nested list of arbitrary depth.",
81
+ "base_output": "</think>\n</think>\n\n\n\nOkay\n\n\n</think>\n\n\n\n\n</think>\n\n...\n\nWe\n</think>\n\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n...\n\n",
82
+ "lora_output": "</think>\n\n<think>\n</think>\n\n```python\ndef flatten(lst):\n result = []\n for item in iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(iter(",
83
+ "base_score": 0,
84
+ "lora_score": 4,
85
+ "improved": true
86
+ },
87
+ {
88
+ "prompt": "Implement a LRU cache in Python.",
89
+ "base_output": "</think>\nOkay\n\n<\nOkay\n<\n</think>\n\n<\n</think>\n\n<\n</think>\n\n\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n<\n...\n</think>\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n",
90
+ "lora_output": "</think>\n\n</think>\n```python\nfrom collections import deque\nfrom typing import Dict\n\nclass LRRAbit:\n def __init__(self,capacity):\n self.capacity = capacity\n self.cache = {}\n self.lbit = \n self.lbit_key = None\n self.lbit_value = None\n self.cache_keys = set()\n self.cache_vals = {}\n self.cache_lbits = deque.QueueType(dict)\n self.cache_lkey = deque.QueueType(dict)\n self.cache_lkey_key = deque.QueueType(dict)\n self.cache_lkey_value = deque.QueueType(dict)\n self.cache_lkey_key_key = deque.QueueType(dict)\n self.cache_lkey_key_value = deque.QueueType(dict)\n self.cache_lkey_key_key_key = deque.QueueType(dict)\n self.cache_lkey_key_value_key = deque.QueueType(dict)\n self.cache_lkey_key_",
91
+ "base_score": 0,
92
+ "lora_score": 5,
93
+ "improved": true
94
+ }
95
+ ],
96
+ "improved_count": 10,
97
+ "total": 10,
98
+ "improvement_rate": 1.0,
99
+ "avg_base_score": 0.5,
100
+ "avg_lora_score": 4.6,
101
+ "pct_increase": 819.9999999999999,
102
+ "passed_20pct": true
103
+ }