ThaiLLM-8B-ThaiLaw

A fine-tuned version of ThaiLLM/ThaiLLM-8B specialized for Thai legal documents and law-related tasks.

Note : After testing I find that it hallucinate so badly that I can't even recommend anyone using this model. I promised that further model release will be better quality.

Model Description

• Base Model: ThaiLLM/ThaiLLM-8B (Qwen3 architecture)
• Training Data: Thai legal documents from iapp/rag_thai_laws
• Language: Thai (th)
• License: Apache 2.0
• Training Method: QLoRA (4-bit quantization)

Training Details

Dataset

This model was fine-tuned exclusively on:

• iapp/rag_thai_laws (42,755 samples)
  • Thai law texts from the Office of the Council of State
  • Cleaned and processed dataset
  • Contains various types of Thai legal documents

Hardware Configuration

• GPU: 2x NVIDIA RTX PRO 4000 (24GB VRAM each)
• Training Time: ~2 hours
• Framework: PyTorch with Unsloth optimizations

Training Hyperparameters

• Method: QLoRA (4-bit quantization)
• Max Sequence Length: 2048
• Batch Size: 4 per device
• Gradient Accumulation Steps: 8 (effective batch size: 64)
• LoRA Rank (r): 16
• LoRA Alpha: 32
• LoRA Dropout: 0.05
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Learning Rate: 2e-4
• Embedding Learning Rate: 1e-5
• Epochs: 1
• Optimizer: adamw_8bit
• Warmup Ratio: 0.05
• Data Packing: Enabled (2-3x speedup)
• Gradient Checkpointing: Enabled

Training Procedure

The model was trained using Unsloth's efficient QLoRA implementation with the following optimizations:

• 4-bit quantization for memory efficiency
• Data packing for faster training on short texts
• Gradient checkpointing to reduce memory usage
• ChatML template for instruction-following format

Usage

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "sthaps/ThaiLLM-8B-ThaiLaw"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Example usage
messages = [
    {"role": "system", "content": "คุณเป็นผู้ช่วยด้านกฎหมายไทยที่เชี่ยวชาญ คุณต้องตอบคำถามเกี่ยวกับกฎหมายไทยอย่างถูกต้องและครบถ้วน"},
    {"role": "user", "content": "อธิบายเกี่ยวกับพระราชบัญญัติ"},
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, top_p=0.95)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Using with Unsloth (Recommended)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="sthaps/ThaiLLM-8B-ThaiLaw",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)

# Enable faster inference
FastLanguageModel.for_inference(model)

messages = [
    {"role": "system", "content": "คุณเป็นผู้ช่วยด้านกฎหมายไทยที่เชี่ยวชาญ"},
    {"role": "user", "content": "อธิบายเกี่ยวกับกฎหมายแรงงานไทย"},
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Limitations

• Training Data Scope: The model was trained on a single dataset (iapp/rag_thai_laws) and may not represent all areas of Thai law
• Accuracy: While trained on legal documents, this is an AI model and may produce incorrect information
• Professional Advice: This model should not be used as a substitute for professional legal advice
• Language: Primarily optimized for Thai language legal documents
• Updates: The model does not have access to real-time legal updates or changes

Model Architecture

• Base Architecture: ThaiLLM (8B parameters)
• Fine-tuning Method: QLoRA (Low-Rank Adaptation with 4-bit quantization)
• Trainable Parameters: ~0.5% of total parameters (LoRA adapters only)
• Total Parameters: ~8 billion

Training Metrics

• Training Samples: 42,755
• Training Steps: ~1,336 (with packing)
• Effective Batch Size: 64
• Training Loss: Decreased steadily throughout training
• Convergence: Achieved in 1 epoch with packing

Acknowledgments

• Base Model: ThaiLLM/ThaiLLM-8B by ThaiLLM team
• Dataset: iapp/rag_thai_laws by iAPP
• Training Framework: Unsloth for efficient fine-tuning