TinyTalk / NOTICE.md
TheREZOR's picture
TinyTalk release
61c79ad
|
Raw
History Blame Contribute Delete
2.37 kB

Third-party components and attributions

The source code in this repository is MIT-licensed (see LICENSE). The embedded model and tokenizer artifacts derive from the following works:

Base model — TinyStories-Instruct-3M

  • Ronen Eldan & Yuanzhi Li, TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (arXiv:2305.07759).
  • Weights: https://huggingface.co/roneneldan/TinyStories-Instruct-3M — published without an explicit license tag. The companion TinyStories dataset is CDLA-Sharing-1.0, which places no restrictions on results (e.g. trained models). If the licensing of the base weights matters for your use case, contact the model author.

Fine-tuning data

Tokenizer

  • GPT-2 byte-level BPE vocabulary and merges (vocab.json / merges.txt), from OpenAI's GPT-2 release (https://github.com/openai/gpt-2), MIT (Modified MIT License, Copyright (c) 2019 OpenAI). Embedded here in pruned, re-encoded form inside tok_data.cpp.

Alternative model (not embedded by default)

Vendored keyboard driver (main/keyboard/)

Acknowledgements

  • The inference engine follows the structure of Andrej Karpathy's llama2.c (https://github.com/karpathy/llama2.c), MIT. The code here is an independent implementation extended with Q4_0 quantization, a GPT-Neo forward path, int8 KV cache, and a flash-walking tokenizer.