language:
- ja
library_name: pytorch
pipeline_tag: text-generation
datasets:
- KeisukeMiyamoto/lambda-corpus
- KeisukeMiyamoto/SyntheticTextbook-jp
- KeisukeMiyamoto/SyntheticTalk-jp
base_model: KeisukeMiyamoto/lambda-1-360m-mid
base_model_relation: finetune
tags:
- lambda
- pytorch
- causal-lm
- text-generation
- conversational
- instruction-tuned
- supervised-fine-tuning
- decoder-only
- custom-code
- grouped-query-attention
- rotary-position-embedding
- byte-level-bpe
lambda-1-360m-it
lambda-1-360m-it is an experimental Japanese instruction-tuned language model based on lambda-1-360m-mid and supervised fine-tuned on a synthetic Japanese conversation corpus.
All training code is publicly available at KeisukeMiyamoto1324/lambda.
Model Details
| Item | Value |
|---|---|
| Parameters | 359.9M |
| Architecture | Decoder-only Transformer |
| Context length | 1,024 tokens |
| Tokenizer | Byte-level BPE |
| Vocabulary size | 65,536 |
| Layers | 32 |
| Hidden size | 960 |
| Attention heads | 15 |
| Key-value heads | 5 |
| FFN size | 4,096 |
Training Data
The base model was pretrained on KeisukeMiyamoto/lambda-corpus and further pretrained on KeisukeMiyamoto/SyntheticTextbook-jp.
This model was then supervised fine-tuned on KeisukeMiyamoto/SyntheticTalk-jp.
Usage
git clone https://github.com/KeisukeMiyamoto1324/lambda.git
cd lambda
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
python3 src/inference_it/inference.py \
--model-dir "KeisukeMiyamoto/lambda-1-360m-it" \
--prompt "人工知能とは何ですか" \
--max-new-tokens 128
Example Outputs
| Prompt | Model output |
|---|---|
| 万有引力の法則とはなんですか | すべての物体は互いに引き付け合っており、その力は質量の積に比例し距離の2乗に反比例するという法則のことです。 |
| 日本の首都はどこですか | 東京です。 |
| 北朝鮮の正式名称を教えて | 朝鮮民主主義人民共和国です。 |
| 機械学習とは | コンピュータに大量のデータを学習させ、そこから予測や判断を行う技術のことです。 |
| カレー基本的な材料を教えて | 玉ねぎ、人参とジャガイモを使い、水を入れて煮込みます。スパイスを混ぜて炒めます。 |
| お正月とは何か教えて | 新年になって初めて行う一年の節のことです。神社や寺院などの行事を通じて、新しい年の幸福を願う行事を指します。 |
| 自己紹介して下さい | 私はAIアシスタントです。あなたの好きなことや得意なことを教えてください。一緒に何かしませんか?よろしくお願いします。 |
| おすすめの和食を3つ教えて | 寿司、天ぷら、ラーメンをおすすめします。 |
Limitations
This model is not safety-aligned. It may generate incorrect, biased, unsafe, or low-quality text.
The model was trained primarily on synthetic Japanese text and has not been evaluated on standard benchmarks.
Support Lambda
Lambda is an open-source project for building small Japanese language models from scratch. As a student, I have funded this project with income from my part-time job, but the growing training costs are becoming difficult to cover.
Your support helps cover GPU costs and develop larger models. Thank you for helping Lambda continue to grow.
Vast.ai
Vast.ai offers affordable cloud GPUs for AI training, with NVIDIA H100 SXM GPUs available from around $1.54 per hour. If you purchase credits through the link below, I receive 3% in GPU credits at no extra cost to you.
https://cloud.vast.ai/?ref_id=521936
Ko-fi
Support Lambda with a donation starting from $5.
