---
language:
- ja
library_name: pytorch
pipeline_tag: text-generation
datasets:
- KeisukeMiyamoto/lambda-corpus
- KeisukeMiyamoto/SyntheticTextbook-jp
- KeisukeMiyamoto/SyntheticTalk-jp
base_model: KeisukeMiyamoto/lambda-1-360m-mid
base_model_relation: finetune
tags:
- lambda
- pytorch
- causal-lm
- text-generation
- conversational
- instruction-tuned
- supervised-fine-tuning
- decoder-only
- custom-code
- grouped-query-attention
- rotary-position-embedding
- byte-level-bpe
---

# lambda-1-360m-it

lambda-1-360m-it is an experimental Japanese instruction-tuned language model based on [`lambda-1-360m-mid`](https://huggingface.co/KeisukeMiyamoto/lambda-1-360m-mid) and supervised fine-tuned on a synthetic Japanese conversation corpus.

All training code is publicly available at [KeisukeMiyamoto1324/lambda](https://github.com/KeisukeMiyamoto1324/lambda).

## Model Details

| Item | Value |
|---|---:|
| Parameters | 359.9M |
| Architecture | Decoder-only Transformer |
| Context length | 1,024 tokens |
| Tokenizer | Byte-level BPE |
| Vocabulary size | 65,536 |
| Layers | 32 |
| Hidden size | 960 |
| Attention heads | 15 |
| Key-value heads | 5 |
| FFN size | 4,096 |

## Training Data

The base model was pretrained on [`KeisukeMiyamoto/lambda-corpus`](https://huggingface.co/datasets/KeisukeMiyamoto/lambda-corpus) and further pretrained on [`KeisukeMiyamoto/SyntheticTextbook-jp`](https://huggingface.co/datasets/KeisukeMiyamoto/SyntheticTextbook-jp).

This model was then supervised fine-tuned on [`KeisukeMiyamoto/SyntheticTalk-jp`](https://huggingface.co/datasets/KeisukeMiyamoto/SyntheticTalk-jp).

## Usage

```bash
git clone https://github.com/KeisukeMiyamoto1324/lambda.git
cd lambda
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt

python3 src/inference_it/inference.py \
  --model-dir "KeisukeMiyamoto/lambda-1-360m-it" \
  --prompt "人工知能とは何ですか" \
  --max-new-tokens 128
```

## Example Outputs

| Prompt | Model output |
|---|---|
| 万有引力の法則とはなんですか | すべての物体は互いに引き付け合っており、その力は質量の積に比例し距離の2乗に反比例するという法則のことです。 |
| 日本の首都はどこですか | 東京です。 |
| 北朝鮮の正式名称を教えて | 朝鮮民主主義人民共和国です。 |
| 機械学習とは | コンピュータに大量のデータを学習させ、そこから予測や判断を行う技術のことです。 |
| カレー基本的な材料を教えて | 玉ねぎ、人参とジャガイモを使い、水を入れて煮込みます。スパイスを混ぜて炒めます。 |
| お正月とは何か教えて | 新年になって初めて行う一年の節のことです。神社や寺院などの行事を通じて、新しい年の幸福を願う行事を指します。 |
| 自己紹介して下さい | 私はAIアシスタントです。あなたの好きなことや得意なことを教えてください。一緒に何かしませんか?よろしくお願いします。 |
| おすすめの和食を3つ教えて | 寿司、天ぷら、ラーメンをおすすめします。 |

## Limitations

This model is not safety-aligned. It may generate incorrect, biased, unsafe, or low-quality text.

The model was trained primarily on synthetic Japanese text and has not been evaluated on standard benchmarks.

---

## Support Lambda

[Lambda](https://github.com/KeisukeMiyamoto1324/lambda) is an open-source project for building small Japanese language models from scratch. As a student, I have funded this project with income from my part-time job, but the growing training costs are becoming difficult to cover.

Your support helps cover GPU costs and develop larger models. Thank you for helping Lambda continue to grow.

### Vast.ai

Vast.ai offers affordable cloud GPUs for AI training, with **NVIDIA H100 SXM GPUs available from around $1.54 per hour**. If you purchase credits through the link below, I receive 3% in GPU credits at no extra cost to you.

https://cloud.vast.ai/?ref_id=521936

### Ko-fi

Support Lambda with a donation starting from $5.

<a href="https://ko-fi.com/lambda_llm">
  <img src="assets/support_me_on_kofi_badge_blue.png" alt="Support Lambda on Ko-fi" width="240">
</a>