Hugging Face's logo

Asilarkness
/
testgeniy

Text Generation
Transformers
Safetensors
English
testgeniy
causal-lm
reasoning
mathematics
logic
long-context
4k-context
small-language-model
Model card Files
xet
 Community
Asilarkness commited on
Commit
cd59bf4
·
verified ·
1 Parent(s): 2277b90

Release validated TestGeniy 4K context model as main

Browse files
Files changed (4) hide show
  1. README.md +54 -5
  2. config.json +31 -4
  3. manifest.json +42 -0
  4. model.safetensors +2 -2
README.md CHANGED
@@ -1,9 +1,58 @@
1
  ---
2
- license: other
3
- library_name: pytorch
 
 
 
 
 
 
 
 
 
 
 
 
4
  ---
5
 
6
- # TestGeniy 500M
7
 
8
- 1280d / 24L / GQA 10:2
9
- 3 local + 1 full NoPE, SwiGLU, QK-norm, tied emb, digit tokens.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - en
4
+ library_name: transformers
5
+ pipeline_tag: text-generation
6
+ tags:
7
+ - testgeniy
8
+ - text-generation
9
+ - causal-lm
10
+ - reasoning
11
+ - mathematics
12
+ - logic
13
+ - long-context
14
+ - 4k-context
15
+ - small-language-model
16
  ---
17
 
18
+ # TestGeniy 4K Context Reasoning Model
19
 
20
+ TestGeniy is a compact causal language model focused on mathematical reasoning, formal logic, and helpful text interaction.
21
+
22
+ This `main` release is the validated 4K-context anchor. It is the safe production checkpoint after context-extension and regression testing.
23
+
24
+ ## Release summary
25
+
26
+ - Context window: 4096 tokens.
27
+ - RoPE: extended from 2048 to 4096 positions using the original theta value 500000.
28
+ - Attention: sliding attention with block size 1024 and global attention in layers 3, 7, 11, 15, 19, and 23.
29
+ - Weights: validated `logic_small_scope_step080` anchor, with context buffers extended to 4096.
30
+ - Evaluation questions were kept out of training.
31
+ - This main release does not include the rejected synthetic-CoT candidates.
32
+
33
+ ## Validation
34
+
35
+ The 4K model remained finite on full 4096-token forward passes and answered a 3157-token long-context probe correctly.
36
+
37
+ Fixed paired reasoning gate, 12 examples per dataset:
38
+
39
+ | Benchmark | Anchor | 4K main |
40
+ |---|---:|---:|
41
+ | GSM8K | 2/12 | 2/12 |
42
+ | MATH-500 | 2/12 | 2/12 |
43
+ | ARC-Challenge | 5/12 | 5/12 |
44
+ | FOLIO | 4/12 | 4/12 |
45
+
46
+ The release is a verified context-capability improvement with no measured regression on this gate. It is not presented as a benchmark-accuracy improvement.
47
+
48
+ ## Intended use
49
+
50
+ Use this checkpoint for compact English reasoning experiments, long-context prompting up to 4096 tokens, and further controlled fine-tuning.
51
+
52
+ ## Limitations
53
+
54
+ This is a small research model. It can produce incorrect reasoning or answers, especially on difficult mathematics and formal logic. The benchmark gate above is a regression gate, not a broad capability estimate.
55
+
56
+ ## Provenance
57
+
58
+ Base checkpoint: `logic_small_scope_step080` from this project. The published weights contain no benchmark questions and no synthetic-CoT training data.
config.json CHANGED
@@ -3,15 +3,42 @@
3
  "TestGeniyForCausalLM"
4
  ],
5
  "model_type": "testgeniy",
6
- "vocab_size": 49216,
7
  "hidden_size": 1280,
8
- "intermediate_size": 3584,
9
  "num_hidden_layers": 24,
10
  "num_attention_heads": 10,
11
  "num_key_value_heads": 2,
 
12
  "head_dim": 128,
13
- "max_position_embeddings": 2048,
14
  "rope_theta": 500000.0,
15
  "local_block_size": 1024,
16
- "tie_word_embeddings": true
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
17
  }
 
3
  "TestGeniyForCausalLM"
4
  ],
5
  "model_type": "testgeniy",
6
+ "vocab_size": 0,
7
  "hidden_size": 1280,
 
8
  "num_hidden_layers": 24,
9
  "num_attention_heads": 10,
10
  "num_key_value_heads": 2,
11
+ "intermediate_size": 3584,
12
  "head_dim": 128,
13
+ "max_position_embeddings": 4096,
14
  "rope_theta": 500000.0,
15
  "local_block_size": 1024,
16
+ "layer_types": [
17
+ "sliding_attention",
18
+ "sliding_attention",
19
+ "sliding_attention",
20
+ "full_attention",
21
+ "sliding_attention",
22
+ "sliding_attention",
23
+ "sliding_attention",
24
+ "full_attention",
25
+ "sliding_attention",
26
+ "sliding_attention",
27
+ "sliding_attention",
28
+ "full_attention",
29
+ "sliding_attention",
30
+ "sliding_attention",
31
+ "sliding_attention",
32
+ "full_attention",
33
+ "sliding_attention",
34
+ "sliding_attention",
35
+ "sliding_attention",
36
+ "full_attention",
37
+ "sliding_attention",
38
+ "sliding_attention",
39
+ "sliding_attention",
40
+ "full_attention"
41
+ ],
42
+ "base_checkpoint": "logic_small_scope_step080",
43
+ "weights_updated": false
44
  }
manifest.json ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "release": "context4k_main",
3
+ "checkpoint": "logic_small_scope_step080",
4
+ "context_length": 4096,
5
+ "rope_theta": 500000.0,
6
+ "weights_updated": false,
7
+ "benchmark_rows_used": false,
8
+ "synthetic_rows_used": false,
9
+ "gate": {
10
+ "gsm8k": [
11
+ 2,
12
+ 2
13
+ ],
14
+ "math500": [
15
+ 2,
16
+ 2
17
+ ],
18
+ "arc_challenge": [
19
+ 5,
20
+ 5
21
+ ],
22
+ "folio": [
23
+ 4,
24
+ 4
25
+ ]
26
+ },
27
+ "hf_repo": "Asilarkness/testgeniy",
28
+ "hub_path": "",
29
+ "published_as": "main",
30
+ "model_card": "README.md",
31
+ "tags": [
32
+ "testgeniy",
33
+ "text-generation",
34
+ "causal-lm",
35
+ "reasoning",
36
+ "mathematics",
37
+ "logic",
38
+ "long-context",
39
+ "4k-context",
40
+ "small-language-model"
41
+ ]
42
+ }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:8d348ef7338ab4757379749ea6de26ab19031de66a1020b473fd7c649c94cbcf
3
- size 1101624392
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:004d52e0eaf3a40a90173f23e19a54d669b9384faf0473825f282ca17128d0a7
3
+ size 1101624480