README.md

# summerMC/TRM-text-v4

<div align="center">

# TRM-text-v4

Experimental Japanese Reasoning Language Model

</div>

## Overview

**TRM-text-v4** is an experimental Japanese language model developed to explore improved reasoning capabilities and extended context handling.

The model is designed to generate intermediate reasoning traces using `<think>` tags before producing final answers. It combines efficient fine-tuning techniques with experimental approaches to context extension.

> **Status:** Research Preview / Experimental Release

---

## Features

### Japanese Chain-of-Thought Reasoning

TRM-text-v4 is fine-tuned to produce structured reasoning steps in Japanese using `<think>` tags.

Example:

```text
User: 富士山の標高は？
Assistant: <think>
質問は富士山の標高についてである。
一般的に知られている標高を確認する。
富士山の標高は3776メートルである。
</think>
富士山の標高は3,776メートルです。
```

---

### Extended Context Experiments

The model incorporates experimental RoPE scaling techniques to investigate longer-context capabilities beyond the original configuration.

Current implementation focuses on practical extensions while evaluating numerical stability and hardware constraints.

---

### Efficient Fine-Tuning

TRM-text-v4 was trained using parameter-efficient fine-tuning methods:

* 4-bit NF4 quantization via BitsAndBytes
* LoRA adaptation
* Memory-efficient optimization for consumer GPUs

---

## Model Details

| Item           | Value                                  |
| -------------- | -------------------------------------- |
| Model Name     | TRM-text-v4                            |
| Architecture   | MoE-based Causal Language Model        |
| Language       | Japanese                               |
| License        | Apache-2.0 (or specify actual license) |
| Intended Use   | Research and experimentation           |
| Release Status | Experimental                           |

---

## Training Configuration

### Fine-Tuning

* LoRA Rank (`r`): **16**
* LoRA Alpha: **32**
* Target Modules: **All Linear Layers**
* Learning Rate: **3e-4**
* Precision: **FP16**
* Quantization: **4-bit NF4**

### Context Configuration

* RoPE Scaling: **Dynamic**
* RoPE Scaling Factor: **2.0**
* Maximum Position Embeddings: **2048**

> Long-context support beyond this configuration remains under investigation and should be considered experimental.

---

## Datasets

### Instruction Tuning

* **Databricks Dolly-15k-ja**

  * Used to improve Japanese instruction-following ability.

### Reasoning Data

* Reasoning templates inspired by datasets such as:

  * Magpie
  * Sakura Reasoning

> Portions of the reasoning pipeline are experimental and may evolve in future releases.

---

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "summerMC/TRM-text-v4"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

prompt = """User: 人工知能とは？
Assistant: <think>
"""

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

## Prompt Format

Recommended prompt template:

```text
User: [Question]
Assistant: <think>
[Reasoning Process]
</think>
[Final Answer]
```

---

## Limitations

* Reasoning traces may occasionally contain mixed Japanese and English terminology.
* Generated reasoning processes are not guaranteed to reflect the model's actual internal computation.
* Long-context behavior beyond validated ranges may exhibit instability.
* The model may generate incorrect or fabricated information and should not be used in high-risk domains without human verification.

---

## Intended Use

Recommended uses:

* Research on Japanese reasoning language models
* Prompt engineering experiments
* Educational demonstrations
* Evaluation of Chain-of-Thought prompting strategies

Not recommended for:

* Medical decision-making
* Legal advice
* Financial advice
* Safety-critical applications

---

## Citation

```bibtex
@misc{summermc_trm_text_v4,
  title        = {TRM-text-v4: Experimental Japanese Reasoning Language Model},
  author       = {summerMC},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/summerMC/TRM-text-v4}}
}
```

---

## Acknowledgements

TRM-text-v4 builds upon the open-source ecosystem, including:

* Hugging Face Transformers
* PEFT
* BitsAndBytes
* Databricks Dolly
* The broader open-source LLM community

Special thanks to all researchers and contributors advancing efficient Japanese language modeling.