| --- |
| library_name: deepj |
| license: mit |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - deepj |
| --- |
| |
| # DeepJ TinyStories |
|
|
| An experimental 1,301,120-parameter language model trained with [DeepJ](https://github.com/KirstenAli/DeepJ). On 100 deterministic TinyStories validation windows (12,800 tokens), its loss was 2.182273 and perplexity was 8.866. |
|
|
| DeepJ is a Java neural network library with shared tensors, automatic differentiation, and CPU execution with optional Apple Metal acceleration. Its network APIs include feed-forward, convolutional, recurrent, autoencoder, and Transformer families. |
|
|
| ## Model details |
|
|
| This compact language model was created with [**DeepJ**](https://github.com/KirstenAli/DeepJ) and uses its Prism decoder-only Transformer architecture with 4 layers, a hidden size of 128, 4 attention heads, Q rank 64, KV rank 32, and a 2,048-token vocabulary. |
|
|
| Generation uses a request-scoped key/value cache. By default, DeepJ rebuilds it when the context window shifts. Optional [rolling generation](https://github.com/KirstenAli/DeepJ/blob/master/docs/inference.md#rolling-generation) avoids repeated rebuilds, with different attention behaviour after eviction; it does not extend the model's trained context capability. The `model.dj` checkpoint and `tokenizer.bpe` files use DeepJ's versioned binary formats. They are not directly loadable by the Python Transformers library. This older checkpoint predates embedded architecture settings, so its configuration must be supplied when loading it. |
|
|
| ## Usage |
|
|
| Use [DeepJ 0.13.0-alpha](https://github.com/KirstenAli/DeepJ): |
|
|
| ```xml |
| <dependency> |
| <groupId>io.github.kirstenali</groupId> |
| <artifactId>deepj</artifactId> |
| <version>0.13.0-alpha</version> |
| </dependency> |
| ``` |
|
|
| Download this repository's `model.dj` and `tokenizer.bpe` into `downloaded-model`. |
|
|
| ```java |
| import io.github.kirstenali.deepj.nn.transformer.TextGenerator; |
| import io.github.kirstenali.deepj.nn.transformer.prism.DeepJPrism; |
| import io.github.kirstenali.deepj.nn.transformer.prism.DeepJPrismConfig; |
| import io.github.kirstenali.deepj.tokenizers.bpe.BPEModel; |
| import io.github.kirstenali.deepj.tokenizers.bpe.BPEModelIO; |
| import io.github.kirstenali.deepj.tokenizers.bpe.BPETokenizer; |
| import java.nio.file.Path; |
| |
| Path directory = Path.of("downloaded-model"); |
| BPEModel bpe = BPEModelIO.load(directory.resolve("tokenizer.bpe")); |
| BPETokenizer tokenizer = new BPETokenizer(bpe); |
| DeepJPrismConfig config = new DeepJPrismConfig( |
| 2048, 128, 128, 4, 4, 384, 64, 32, 0.2f, 1.0f); |
| DeepJPrism model = new DeepJPrism(config, 42L); |
| model.load(directory.resolve("model.dj")); |
| String text = TextGenerator.generate(model, tokenizer, config, |
| "Once upon a time", 80, 0.8f, 40, 2026L, 1.1f); |
| ``` |
|
|
| ### Generation settings |
|
|
| | Setting | What it changes | |
| |---|---| |
| | Temperature | Lower values favor likely next tokens; higher values give other choices a better chance. | |
| | Top-k | Samples from the top `k` token choices; `0` considers all. | |
| | Max tokens | Limits new tokens, not words. | |
| | Seed | Controls sampling randomness; changing it can change the response. | |
| | Repetition penalty | Makes the model less likely to repeat itself. `1.0` turns it off; try `1.1` if responses loop. | |
|
|
| For a newer self-describing, instruction-tuned model that supports the one-call `DeepJChat.load(directory)` API, desktop UI and [command-line interface](https://github.com/KirstenAli/DeepJ/blob/master/docs/cli.md), see [Frisky 90M](https://huggingface.co/netsrik/frisky-90m). |
|
|
| ## Training data |
|
|
| Trained for 10,000 steps with batch size 1 and 128-token sequences on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories), a synthetic English dataset licensed CDLA-Sharing-1.0. The BPE vocabulary was trained from a bounded sample of the training split. |
|
|
| ## Limitations |
|
|
| This small experimental model was trained only on synthetic children's stories. It may produce incorrect, repetitive, biased, or unsuitable text. It has not been evaluated for safety or downstream use and is not a general-purpose assistant. |
