File size: 1,444 Bytes
9591aa7 57d2740 9591aa7 57d2740 7462a9d 57d2740 116fa4e 57d2740 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | ---
language:
- ja
library_name: transformers
tags:
- myllm
- causal-lm
- custom-code
- safetensors
pipeline_tag: text-generation
---
# lambda-1-160m-base
lambda-1-160m-base is an experimental language model created with a custom `myllm` decoder-only Transformer implementation.
All training code is publicly available at [KeisukeMiyamoto1324/myllm](https://github.com/KeisukeMiyamoto1324/myllm).
## Model Details
| Item | Value |
|---|---:|
| Parameters | 164.5M |
| Architecture | Decoder-only Transformer |
| Context length | 1024 tokens |
| Tokenizer | Byte-level BPE |
| Vocabulary size | 65,536 |
| Layers | 16 |
| Hidden size | 768 |
| Attention heads | 12 |
| FFN size | 3,072 |
## Training Data
The model was pretrained on a Japanese text mixture.
| Dataset | Notes |
|---|---|
| `MK0727/CleanedFineWeb2Edu-jp` | Filtered Japanese web corpus |
| `MK0727/SyntheticTextbook-jp` | Synthetic Japanese corpus |
## Usage
```bash
git clone https://github.com/KeisukeMiyamoto1324/lambda.git
cd lambda
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
python3 src/inference_base/inference_hf.py \
--prompt "人工知能とは" \
--max-new-tokens 64
```
## Limitations
This model is not instruction-tuned or safety-aligned. It may generate incorrect, biased, unsafe, or low-quality text.
The model was trained on a limited Japanese corpus mixture and has not been evaluated on standard benchmarks.
|