File size: 4,301 Bytes
832cdb9 f643652 832cdb9 0d2b33d 832cdb9 b5b22ef cf1a486 b5b22ef 507ae90 b5b22ef 507ae90 b5b22ef cf1a486 b5b22ef 507ae90 b5b22ef 507ae90 b5b22ef 507ae90 b5b22ef 43cecf8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 | ---
language:
- ja
library_name: pytorch
pipeline_tag: text-generation
datasets:
- KeisukeMiyamoto/lambda-corpus
- KeisukeMiyamoto/SyntheticTextbook-jp
- KeisukeMiyamoto/SyntheticTalk-jp
base_model: KeisukeMiyamoto/lambda-1-360m-mid
base_model_relation: finetune
tags:
- lambda
- pytorch
- causal-lm
- text-generation
- conversational
- instruction-tuned
- supervised-fine-tuning
- decoder-only
- custom-code
- grouped-query-attention
- rotary-position-embedding
- byte-level-bpe
---
# lambda-1-360m-it
lambda-1-360m-it is an experimental Japanese instruction-tuned language model based on [`lambda-1-360m-mid`](https://huggingface.co/KeisukeMiyamoto/lambda-1-360m-mid) and supervised fine-tuned on a synthetic Japanese conversation corpus.
All training code is publicly available at [KeisukeMiyamoto1324/lambda](https://github.com/KeisukeMiyamoto1324/lambda).
## Model Details
| Item | Value |
|---|---:|
| Parameters | 359.9M |
| Architecture | Decoder-only Transformer |
| Context length | 1,024 tokens |
| Tokenizer | Byte-level BPE |
| Vocabulary size | 65,536 |
| Layers | 32 |
| Hidden size | 960 |
| Attention heads | 15 |
| Key-value heads | 5 |
| FFN size | 4,096 |
## Training Data
The base model was pretrained on [`KeisukeMiyamoto/lambda-corpus`](https://huggingface.co/datasets/KeisukeMiyamoto/lambda-corpus) and further pretrained on [`KeisukeMiyamoto/SyntheticTextbook-jp`](https://huggingface.co/datasets/KeisukeMiyamoto/SyntheticTextbook-jp).
This model was then supervised fine-tuned on [`KeisukeMiyamoto/SyntheticTalk-jp`](https://huggingface.co/datasets/KeisukeMiyamoto/SyntheticTalk-jp).
## Usage
```bash
git clone https://github.com/KeisukeMiyamoto1324/lambda.git
cd lambda
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
python3 src/inference_it/inference.py \
--model-dir "KeisukeMiyamoto/lambda-1-360m-it" \
--prompt "人工知能とは何ですか" \
--max-new-tokens 128
```
## Example Outputs
| Prompt | Model output |
|---|---|
| 万有引力の法則とはなんですか | すべての物体は互いに引き付け合っており、その力は質量の積に比例し距離の2乗に反比例するという法則のことです。 |
| 日本の首都はどこですか | 東京です。 |
| 北朝鮮の正式名称を教えて | 朝鮮民主主義人民共和国です。 |
| 機械学習とは | コンピュータに大量のデータを学習させ、そこから予測や判断を行う技術のことです。 |
| カレー基本的な材料を教えて | 玉ねぎ、人参とジャガイモを使い、水を入れて煮込みます。スパイスを混ぜて炒めます。 |
| お正月とは何か教えて | 新年になって初めて行う一年の節のことです。神社や寺院などの行事を通じて、新しい年の幸福を願う行事を指します。 |
| 自己紹介して下さい | 私はAIアシスタントです。あなたの好きなことや得意なことを教えてください。一緒に何かしませんか?よろしくお願いします。 |
| おすすめの和食を3つ教えて | 寿司、天ぷら、ラーメンをおすすめします。 |
## Limitations
This model is not safety-aligned. It may generate incorrect, biased, unsafe, or low-quality text.
The model was trained primarily on synthetic Japanese text and has not been evaluated on standard benchmarks.
---
## Support Lambda
[Lambda](https://github.com/KeisukeMiyamoto1324/lambda) is an open-source project for building small Japanese language models from scratch. As a student, I have funded this project with income from my part-time job, but the growing training costs are becoming difficult to cover.
Your support helps cover GPU costs and develop larger models. Thank you for helping Lambda continue to grow.
### Vast.ai
Vast.ai offers affordable cloud GPUs for AI training, with **NVIDIA H100 SXM GPUs available from around $1.54 per hour**. If you purchase credits through the link below, I receive 3% in GPU credits at no extra cost to you.
https://cloud.vast.ai/?ref_id=521936
### Ko-fi
Support Lambda with a donation starting from $5.
<a href="https://ko-fi.com/lambda_llm">
<img src="assets/support_me_on_kofi_badge_blue.png" alt="Support Lambda on Ko-fi" width="240">
</a>
|