Update README.md
Browse files
README.md
CHANGED
|
@@ -10,25 +10,23 @@ tags:
|
|
| 10 |
|
| 11 |
# DeepJ TinyStories
|
| 12 |
|
| 13 |
-
An experimental 1,301,120-parameter language model trained with DeepJ. On 100 deterministic TinyStories validation windows (
|
| 14 |
|
| 15 |
## Model details
|
| 16 |
|
| 17 |
-
This compact language model was created with [DeepJ](https://github.com/KirstenAli/DeepJ) and uses a DeepSeek-style Transformer architecture with 4 layers, a hidden size of 128, 4 attention heads, Q rank 64, KV rank 32 and a 2,048-token vocabulary.
|
| 18 |
|
| 19 |
-
It is not an exact implementation of DeepSeek V2, V3 or R1 and does not
|
| 20 |
-
The `model.dj` checkpoint and `tokenizer.bpe` files use DeepJ's versioned binary formats.
|
| 21 |
-
They are not directly loadable by the Python Transformers library.
|
| 22 |
|
| 23 |
## Usage
|
| 24 |
|
| 25 |
-
Use [DeepJ 0.
|
| 26 |
|
| 27 |
```xml
|
| 28 |
<dependency>
|
| 29 |
<groupId>io.github.kirstenali</groupId>
|
| 30 |
<artifactId>deepj</artifactId>
|
| 31 |
-
<version>0.
|
| 32 |
</dependency>
|
| 33 |
```
|
| 34 |
|
|
@@ -46,10 +44,12 @@ String text = TextGenerator.generate(model, tokenizer, config,
|
|
| 46 |
"Once upon a time", 80, 0.8f, 40, 2026L);
|
| 47 |
```
|
| 48 |
|
|
|
|
|
|
|
| 49 |
## Training data
|
| 50 |
|
| 51 |
-
Trained for
|
| 52 |
|
| 53 |
## Limitations
|
| 54 |
|
| 55 |
-
|
|
|
|
| 10 |
|
| 11 |
# DeepJ TinyStories
|
| 12 |
|
| 13 |
+
An experimental 1,301,120-parameter language model trained with [DeepJ](https://github.com/KirstenAli/DeepJ). On 100 deterministic TinyStories validation windows (12,800 tokens), its loss was 2.182273 and perplexity was 8.866.
|
| 14 |
|
| 15 |
## Model details
|
| 16 |
|
| 17 |
+
This compact language model was created with [**DeepJ**](https://github.com/KirstenAli/DeepJ) and uses a DeepSeek-style Transformer architecture with 4 layers, a hidden size of 128, 4 attention heads, Q rank 64, KV rank 32, and a 2,048-token vocabulary.
|
| 18 |
|
| 19 |
+
It is not an exact implementation of DeepSeek V2, V3, or R1 and does not use an incremental KV cache. The `model.dj` checkpoint and `tokenizer.bpe` files use DeepJ's versioned binary formats. They are not directly loadable by the Python Transformers library. This older checkpoint predates embedded architecture settings, so its configuration must be supplied when loading it.
|
|
|
|
|
|
|
| 20 |
|
| 21 |
## Usage
|
| 22 |
|
| 23 |
+
Use [DeepJ 0.9.4-alpha](https://central.sonatype.com/artifact/io.github.kirstenali/deepj/0.9.4-alpha) or a newer compatible release:
|
| 24 |
|
| 25 |
```xml
|
| 26 |
<dependency>
|
| 27 |
<groupId>io.github.kirstenali</groupId>
|
| 28 |
<artifactId>deepj</artifactId>
|
| 29 |
+
<version>0.9.4-alpha</version>
|
| 30 |
</dependency>
|
| 31 |
```
|
| 32 |
|
|
|
|
| 44 |
"Once upon a time", 80, 0.8f, 40, 2026L);
|
| 45 |
```
|
| 46 |
|
| 47 |
+
For a newer self-describing, instruction-tuned DeepJ model, see [Frisky 90M](https://huggingface.co/netsrik/frisky-90m).
|
| 48 |
+
|
| 49 |
## Training data
|
| 50 |
|
| 51 |
+
Trained for 10,000 steps with batch size 1 and 128-token sequences on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories), a synthetic English dataset licensed CDLA-Sharing-1.0. The BPE vocabulary was trained from a bounded sample of the training split.
|
| 52 |
|
| 53 |
## Limitations
|
| 54 |
|
| 55 |
+
This small experimental model was trained only on synthetic children's stories. It may produce incorrect, repetitive, biased, or unsuitable text. It has not been evaluated for safety or downstream use and is not a general-purpose assistant.
|