browser-use

/

bu-30b-a3b-preview

like 194

Follow

Browser Use 105

Summary

I've identified significant Japanese text quality issues in bu-30b-a3b-preview when used with browser-use's system prompt and DOM state context. Through controlled experiments, I confirmed that the fine-tuned model shows severe degradation in Japanese output quality compared to the base model (Qwen3-VL-30B-A3B-Instruct) under browser-use operating conditions.

Environment

• Inference: llama.cpp (llama-server)
• Quantization: Q8_0 (via bartowski/browser-use_bu-30b-a3b-preview-GGUF)
• Hardware: AMD Radeon Instinct MI25 x4 / NVIDIA Tesla P100 x4
• Context size: 24576-32768 tokens
• Version: Latest as of 2025-12-25

Experiment Design

I conducted three phases of testing to isolate the cause:

Phase 1: Direct API Requests (Baseline)

Simple Japanese text repetition tasks without any system prompt:

• Task: Output "しぐれうい" (Shigure Ui) 5 times
• Task: Output "文鳥と暮らしています" (I live with a Java sparrow) 5 times

Results: Both base model and bu-30b achieve 100% accuracy

Phase 2: Real-world Browser-Use Task

Web browsing task: Search "しぐれうい" on Google, visit 3 sites, extract summaries.
15 test runs with bu-30b-a3b-preview.

Results: Significant Japanese text corruption observed

Phase 3: Controlled Condition Tests

Added browser-use-specific conditions incrementally:

1. Exp1: Add browser-use system prompt only
2. Exp2: Add long context (~3,000 tokens of English text)
3. Exp3: Add DOM extraction text (Japanese Wikipedia-style content)

Results

Accuracy Comparison (5 runs × 5 repetitions = 25 samples per test)

Summary Statistics

Error Patterns Observed

1. Dakuten (Voiced Consonant Mark) Errors

Expected: しぐれうい (Shigure Ui)

2. Kanji Conversion Errors

Expected: 文鳥と暮らしています (I live with a Java sparrow)

3. Completely Garbled Output (from real browser-use runs)

• "222万人の動画を近期的に選み可能なテーンデートの中"
• "イラストレーターと倉画家であるテーシメドの16歳（仮）でお経推きを動いでいる"

These are completely meaningless in Japanese.

Analysis

Key Finding

The base model (Qwen3-VL-30B-A3B-Instruct) maintains 76-100% accuracy across all conditions, while bu-30b-a3b-preview drops to 0-28% when browser-use system prompt and DOM context are added.

This strongly suggests that the fine-tuning process degraded the model's Japanese language capabilities, particularly when operating in the structured output format required by browser-use.

Possible Causes

1. Training data predominantly English: The fine-tuning dataset may have been mostly English browser automation examples
2. JSON output format interference: Training to output structured JSON may have disrupted Japanese token generation
3. Prompt sensitivity: The model may have become overly sensitive to specific prompt structures, causing instability in Japanese generation

Reproduction Steps

1. Load bu-30b-a3b-preview with llama.cpp
2. Use the browser-use system prompt:

You are a browser-use agent. You automate browser tasks by outputting structured JSON actions.
...

3. Add Japanese DOM content to the user message
4. Request Japanese text output
5. Compare with base model (Qwen3-VL-30B-A3B-Instruct) under same conditions

Requests

1. Could you confirm the language distribution of the fine-tuning dataset? Was Japanese (or other non-English languages) included?

2. Are there plans to improve multilingual support? Japanese is widely used in browser automation for Japanese websites.

3. Any recommended workarounds? For example:

   • Using base model with custom prompts?
   • Specific inference parameters that might help?

Appendix: Test Methodology

• Server: llama-server with OpenAI-compatible API
• Temperature: 0.7
• Max tokens: 512
• Test runs: 5 runs per condition, 5 repetitions per run = 25 samples
• Evaluation: Exact string match counting

Thank you for developing this specialized browser automation model! I hope this feedback helps improve multilingual support in future versions.