| --- |
| language: |
| - ja |
| library_name: pytorch |
| pipeline_tag: text-generation |
| datasets: |
| - KeisukeMiyamoto/lambda-corpus |
| - KeisukeMiyamoto/SyntheticTextbook-jp |
| - KeisukeMiyamoto/SyntheticTalk-jp |
| base_model: KeisukeMiyamoto/lambda-1-360m-mid |
| base_model_relation: finetune |
| tags: |
| - lambda |
| - pytorch |
| - causal-lm |
| - text-generation |
| - conversational |
| - instruction-tuned |
| - supervised-fine-tuning |
| - decoder-only |
| - custom-code |
| - grouped-query-attention |
| - rotary-position-embedding |
| - byte-level-bpe |
| --- |
| |
| # lambda-1-360m-it |
|
|
| lambda-1-360m-it is an experimental Japanese instruction-tuned language model based on [`lambda-1-360m-mid`](https://huggingface.co/KeisukeMiyamoto/lambda-1-360m-mid) and supervised fine-tuned on a synthetic Japanese conversation corpus. |
|
|
| All training code is publicly available at [KeisukeMiyamoto1324/lambda](https://github.com/KeisukeMiyamoto1324/lambda). |
|
|
| ## Model Details |
|
|
| | Item | Value | |
| |---|---:| |
| | Parameters | 359.9M | |
| | Architecture | Decoder-only Transformer | |
| | Context length | 1,024 tokens | |
| | Tokenizer | Byte-level BPE | |
| | Vocabulary size | 65,536 | |
| | Layers | 32 | |
| | Hidden size | 960 | |
| | Attention heads | 15 | |
| | Key-value heads | 5 | |
| | FFN size | 4,096 | |
|
|
| ## Training Data |
|
|
| The base model was pretrained on [`KeisukeMiyamoto/lambda-corpus`](https://huggingface.co/datasets/KeisukeMiyamoto/lambda-corpus) and further pretrained on [`KeisukeMiyamoto/SyntheticTextbook-jp`](https://huggingface.co/datasets/KeisukeMiyamoto/SyntheticTextbook-jp). |
|
|
| This model was then supervised fine-tuned on [`KeisukeMiyamoto/SyntheticTalk-jp`](https://huggingface.co/datasets/KeisukeMiyamoto/SyntheticTalk-jp). |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/KeisukeMiyamoto1324/lambda.git |
| cd lambda |
| python3 -m venv venv |
| source venv/bin/activate |
| pip3 install -r requirements.txt |
| |
| python3 src/inference_it/inference.py \ |
| --model-dir "KeisukeMiyamoto/lambda-1-360m-it" \ |
| --prompt "人工知能とは何ですか" \ |
| --max-new-tokens 128 |
| ``` |
|
|
| ## Example Outputs |
|
|
| | Prompt | Model output | |
| |---|---| |
| | 万有引力の法則とはなんですか | すべての物体は互いに引き付け合っており、その力は質量の積に比例し距離の2乗に反比例するという法則のことです。 | |
| | 日本の首都はどこですか | 東京です。 | |
| | 北朝鮮の正式名称を教えて | 朝鮮民主主義人民共和国です。 | |
| | 機械学習とは | コンピュータに大量のデータを学習させ、そこから予測や判断を行う技術のことです。 | |
| | カレー基本的な材料を教えて | 玉ねぎ、人参とジャガイモを使い、水を入れて煮込みます。スパイスを混ぜて炒めます。 | |
| | お正月とは何か教えて | 新年になって初めて行う一年の節のことです。神社や寺院などの行事を通じて、新しい年の幸福を願う行事を指します。 | |
| | 自己紹介して下さい | 私はAIアシスタントです。あなたの好きなことや得意なことを教えてください。一緒に何かしませんか?よろしくお願いします。 | |
| | おすすめの和食を3つ教えて | 寿司、天ぷら、ラーメンをおすすめします。 | |
|
|
| ## Limitations |
|
|
| This model is not safety-aligned. It may generate incorrect, biased, unsafe, or low-quality text. |
|
|
| The model was trained primarily on synthetic Japanese text and has not been evaluated on standard benchmarks. |
|
|
| --- |
|
|
| ## Support Lambda |
|
|
| [Lambda](https://github.com/KeisukeMiyamoto1324/lambda) is an open-source project for building small Japanese language models from scratch. As a student, I have funded this project with income from my part-time job, but the growing training costs are becoming difficult to cover. |
|
|
| Your support helps cover GPU costs and develop larger models. Thank you for helping Lambda continue to grow. |
|
|
| ### Vast.ai |
|
|
| Vast.ai offers affordable cloud GPUs for AI training, with **NVIDIA H100 SXM GPUs available from around $1.54 per hour**. If you purchase credits through the link below, I receive 3% in GPU credits at no extra cost to you. |
|
|
| https://cloud.vast.ai/?ref_id=521936 |
| |
| ### Ko-fi |
| |
| Support Lambda with a donation starting from $5. |
| |
| <a href="https://ko-fi.com/lambda_llm"> |
| <img src="assets/support_me_on_kofi_badge_blue.png" alt="Support Lambda on Ko-fi" width="240"> |
| </a> |
| |
