README.md

---
language:
- ja
library_name: pytorch
pipeline_tag: text-generation
datasets:
- KeisukeMiyamoto/lambda-corpus
tags:
- lambda
- pytorch
- causal-lm
- text-generation
- decoder-only
- custom-code
- grouped-query-attention
- rotary-position-embedding
- byte-level-bpe
- pretrained
---

# lambda-1-360m-base

lambda-1-360m-base is an experimental Japanese language model created with a custom decoder-only Transformer implementation.

All training code is publicly available at [KeisukeMiyamoto1324/lambda](https://github.com/KeisukeMiyamoto1324/lambda).

## Model Details

| Item | Value |
|---|---:|
| Parameters | 359.9M |
| Architecture | Decoder-only Transformer |
| Context length | 1,024 tokens |
| Tokenizer | Byte-level BPE |
| Vocabulary size | 65,536 |
| Layers | 32 |
| Hidden size | 960 |
| Attention heads | 15 |
| Key-value heads | 5 |
| FFN size | 4,096 |

## Training Data

The model was pretrained on the Japanese text corpus [`KeisukeMiyamoto/lambda-corpus`](https://huggingface.co/datasets/KeisukeMiyamoto/lambda-corpus).

## Usage

```bash
git clone https://github.com/KeisukeMiyamoto1324/lambda.git
cd lambda
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt

python3 src/inference_base/inference_hf.py \
  --model-dir "KeisukeMiyamoto/lambda-1-360m-base" \
  --prompt "人工知能とは" \
  --max-new-tokens 64
```

## Limitations

This model is not instruction-tuned or safety-aligned. It may generate incorrect, biased, unsafe, or low-quality text.

The model was trained primarily on Japanese text and has not been evaluated on standard benchmarks.

---

## Support Lambda

[Lambda](https://github.com/KeisukeMiyamoto1324/lambda) is an open-source project for building small Japanese language models from scratch. As a student, I have funded this project with income from my part-time job, but the growing training costs are becoming difficult to cover.

Your support helps cover GPU costs and develop larger models. Thank you for helping Lambda continue to grow.

### Vast.ai

Vast.ai offers affordable cloud GPUs for AI training, with **NVIDIA H100 SXM GPUs available from around $1.54 per hour**. If you purchase credits through the link below, I receive 3% in GPU credits at no extra cost to you.

https://cloud.vast.ai/?ref_id=521936

### Ko-fi

Support Lambda with a donation starting from $5.

<a href="https://ko-fi.com/lambda_llm">
  <img src="assets/support_me_on_kofi_badge_blue.png" alt="Support Lambda on Ko-fi" width="240">
</a>