docs/walkthrough.md

SignBridge — technical walkthrough

Internal technical record of the build. Not a submission deliverable (Build-in-Public extra challenge was dropped on 2026-05-07). Kept around because it documents the AMD-specific engineering thinking and is useful if anyone later asks "why these design choices?".

What we built

A real-time webcam-based ASL → English speech translator. A deaf user signs into the webcam; the pipeline (MediaPipe Hand → trained MLP for static fingerspelling, OR webcam-clip → ffmpeg → fine-tuned Qwen3-VL-8B native video → Qwen3-8B composer → gTTS) returns spoken English in under 2 seconds. Designed to fit Track 3 (Vision & Multimodal AI) with both LLMs running concurrently on a single AMD Instinct MI300X.

Why AMD MI300X

• 192 GB HBM3 — the trained MLP classifier (478 KB), fine-tuned Qwen3-VL-8B (16 GB FP16), Qwen3-8B composer (16 GB FP16), and (V2 stretch) Whisper-large-v3 (3 GB) all fit concurrently with margin for KV cache.
• 5.3 TB/s memory bandwidth — bandwidth-bound streaming workload (many small inferences per second on the MLP + LLM next-token + Qwen3-VL vision encoder) is exactly what bandwidth wins.

Architecture

Snapshot tab (fingerspelling):
  webcam frame → MediaPipe Hand → trained MLP classifier
                   (CPU-fast)        (PyTorch on CPU, ~50 ms)

Record sign tab (motion words):
  webcam recording → ffmpeg (480p, 8 fps, ≤4 s, H.264)
                          ↓
                   vLLM video_url block on AMD MI300X port 8000
                          ↓
              fine-tuned Qwen3-VL-8B (native video understanding)

Both paths converge:
                                   ↓
                          Qwen3-8B sentence composer
                          (vLLM on MI300X port 8001)
                                   ↓
                                  gTTS
                          (Google free TTS, MP3)

Models

Component	Source	Notes
Hand-pose extractor	MediaPipe HandLandmarker (Google)	CPU-only, ~50ms/frame — runs on the HF Space CPU
Static-letter classifier (Snapshot tab)	trained-from-scratch MLP on hand-landmark vectors → 26 ASL letters	3-layer MLP (63→256→256→128→26), 5K trainable params, GELU+dropout. 88.0% test accuracy on a 1,727-image holdout, 90.4% on the gold set. Weights at huggingface.co/LucasLooTan/signbridge-asl-classifier
Motion-sign + fallback recognizer	fine-tuned Qwen/Qwen3-VL-8B-Instruct	LoRA fine-tune on AMD MI300X (rank 16, target q/k/v/o, 2 epochs, 54 min wall-clock on a single MI300X). Eval loss 0.48, transformers gold-set accuracy 92.3%. Merged adapter pushed to huggingface.co/LucasLooTan/signbridge-qwen3vl-8b-asl (17.5 GB)
Sentence composer	Qwen/Qwen3-8B	Pulled from HF Hub; served on MI300X via vLLM. Used for every Speak click — AMD is in the critical path
Text-to-speech	gTTS (Google's free TTS)	Tiny dependency, no model download, MP3 output in <1 s. Coqui XTTS-v2 path is preserved as Tier 1 fallback when installed locally

Datasets

• Marxulia/asl_sign_languages_alphabets_v03 (HF Hub) — 10,873 photographic ASL letter samples; we extracted MediaPipe landmarks (8,639 hands detected) + used the same images for the LoRA fine-tune (9,786 train / 1,087 eval split)
• WLASL Top-100 subset — referenced for V2 motion-sign training (not used in V1)

ROCm / AMD Developer Cloud experience

Day 1 — environment + sanity

• Provisioned an MI300X-1× droplet (192 GB HBM3, 240 GB RAM, 5 TB scratch) at $1.99/hr in ATL1
• Selected the prebuilt vLLM 0.17.1 / ROCm 7.2 Quick-Start image — saved ~30 min vs hand-installing
• ROCm reported the GPU correctly via rocm-smi; vLLM spun up Qwen3-VL-32B + Qwen3-8B in parallel within 12 minutes
• One real friction: vLLM's default 0.0.0.0 binding tripped a Gloo/NCCL error on the host's NIC; fixed by setting VLLM_HOST_IP=127.0.0.1 and GLOO_SOCKET_IFNAME=lo env vars

Day 2 — fine-tuning Qwen3-VL-8B with LoRA on MI300X

• Used the AMD-provided rocm:latest Docker image — torch 2.9.1+ROCm, transformers 4.57.6, peft 0.18.1, accelerate 1.13.0 all preinstalled
• LoRA rank 16 on q/k/v/o projections, FP16, gradient checkpointing with use_reentrant=False
• Critical fix for PEFT + grad-checkpoint: call model.enable_input_require_grads() BEFORE wrapping in PEFT (without it, training stalls at step 0 with "None of the inputs have requires_grad=True")
• 1,224 steps × 4×4 effective batch = 9,786 samples × 2 epochs in 54 minutes; eval loss 0.48
• Spent ~$2 of the $100 credit on this single fine-tune

Day 3 — serving + accuracy comparison

• Three approaches benchmarked on the same 52-image gold set:
  • Qwen3-VL-32B zero-shot: 19.2% — VLMs without ASL-specific tuning struggle with subtle hand shapes
  • MediaPipe + 5K-param MLP: 90.4% — the textbook approach for static pose classification still wins for cost/accuracy ratio
  • LoRA-tuned Qwen3-VL-8B (transformers eval): 92.3% — best, but 4× slower per inference
• Hybrid pipeline ships: MediaPipe+MLP for typical fingerspelling (50ms, ~90%), fine-tuned VLM for motion signs and as fallback when no hand is detected
• Latency on MI300X: Qwen3-8B composer ~0.5s/call, fine-tuned 8B vision recognizer ~1.3s/call

What worked well

• AMD Developer Cloud provisioning was 5 min from "approved" to SSH — credit landed via email and the Quick-Start vLLM image meant zero ROCm setup pain
• 192 GB HBM3 hosted both the 32B vision model and the 8B composer concurrently (gpu-mem 0.55 + 0.30) with margin for KV cache
• Fine-tuning + inference + composing on a single MI300X with no swapping or reloading — the multi-tenant story is real
• The rocm:latest Docker image had the entire training stack (torch, transformers, peft, accelerate, datasets) preinstalled and tested

What we'd flag as friction

• vLLM 0.17.1's image-preprocessing for Qwen3-VL doesn't exactly match transformers' processor — the LoRA-tuned model that scored 92.3% in transformers eval drops to 63.5% via the OpenAI-compatible vLLM endpoint. This is upstream and not AMD-specific, but it limited how aggressively we could lean on the fine-tune for the live demo
• The low-power state warning in rocm-smi while the GPU was idle was cosmetic but confusing — clarifying that "low-power" doesn't mean "stalled" would help first-time users
• Setting VLLM_HOST_IP=127.0.0.1 for single-GPU vLLM on a Gloo backend isn't documented in the AMD vLLM Quick-Start; we found it from a vLLM GitHub issue

What we'd flag as friction

TODO

Why AMD MI300X — concretely

The pipeline (MediaPipe Hand + fine-tuned Qwen3-VL-8B + Qwen3-8B composer + gTTS) fits comfortably on a single MI300X with KV-cache headroom. The same workload on NVIDIA forces sharding once we add the V2 reasoner.

Component	Weights (FP16)	MI300X 1× (192 GB)	H100 80 GB	H200 141 GB
Fine-tuned Qwen3-VL-8B (vision, native video)	~16 GB	✅ fits	✅	✅
Qwen3-8B (composer)	~16 GB	✅ fits	✅	✅
Whisper-large-v3 (V2 reverse direction)	~3 GB	✅ fits	⚠ tight	✅
gTTS (no GPU footprint — Python-side cloud call)	n/a	✅	✅	✅
(V2) Llama-3.1-70B FP8 reasoner upgrade	~70 GB	✅ still fits	❌ doesn't fit at all	⚠ FP8 only, no headroom
Concurrent serving + KV cache	✅ comfortable	❌ requires sharding	⚠ tight	✅

The single-GPU concurrency story is the AMD pitch. V1 fits anywhere; the architecture has clear MI300X headroom for V2 model upgrades that NVIDIA H100 cannot match without sharding across multiple cards.

Deployment ethics

SignBridge is a substrate, not a finished product. We ship the open-source multi-modal pipeline so Deaf-led organisations — schools-for-the-Deaf, regional NGOs, ministries of social services — can deploy on their own AMD compute, fine-tune for their dialect, and own the deployment.

Three principles, drawn from the Deaf-led literature on sign-language AI:

1. ASL only V1 is a scope decision. Sign languages are not interchangeable — BSL, ISL, MSL, CSL, and 200+ other sign languages each deserve their own teams, training data, and Deaf community leadership. Bragg et al. "Systemic Biases in Sign Language AI Research" (2024, Deaf-led position paper) is direct on this point.

2. Deaf community engagement before deployment. Per the ACM ASSETS 2025 "Exploring Collaboration to Center the Deaf Community in Sign Language AI" the productive ML/Deaf collaboration question isn't "how do we build this?" but "should we build this, for whom, with whom?". Any deployment downstream of this code must answer that locally.

3. Privacy by default. SignBridge sessions are ephemeral — webcam frames and audio are processed in-memory and not persisted server-side beyond the request lifetime. In the spirit of Privacy-Aware Sign Language Translation at Scale (ACL 2024).

Future work — academic foundations we'd build on next

• SignCLIP (EMNLP 2024) — learned text↔sign embeddings; replaces the prompt-only composer with a CLIP-style alignment head for higher-quality sign-to-English mapping.
• SL-SLR (arXiv 2509.05188v1) — self-supervised representation learning with motion-aware data augmentation; the right path if we ever train a custom classifier on raw signer footage.
• Continuous SLT trained models (Swin-MSTP, Stack Transformer) — the current trained-from-scratch ceiling on WLASL is ~93.5% Top-1. The VLM zero-shot path we ship here is a deployment-cost play, not an accuracy-ceiling play; SignCLIP-style learned embeddings are the natural V2 step toward that ceiling.

Latency

Target: ≤ 2 s from end-of-sign to start of speech.

Measured on a single MI300X (Day 3):

• MediaPipe Hand detection per frame: ~50 ms (CPU)
• Trained MLP per landmark vector: ~5 ms (CPU)
• Fine-tuned Qwen3-VL-8B per recording (native video, ~1680 prompt tokens): ~1-2 s
• Qwen3-8B sentence composition (≤ 30 tokens): ~300 ms
• gTTS first-audio-chunk: ~500 ms (single round-trip to Google)

MI300X vs NVIDIA H100 — the AMD pitch

Item	MI300X (1 GPU)	H100 (1 GPU)	H100 cluster needed
Fine-tuned Qwen3-VL-8B + Qwen3-8B (both FP16)	✅ fits with margin	⚠️ tight (~32 GB)	maybe 1×, no headroom
+ Whisper-large-v3 + MLP classifier	✅ all concurrent	⚠️ tight (~35 GB total + KV)	likely 1× but no headroom
+ 70B reasoner upgrade (V2)	✅ 70B FP8 ~70 GB still fits	❌ doesn't fit at all	≥3×

The single-GPU concurrency story is the AMD pitch. This V1 fits on H100; the architecture has clear headroom on MI300X for higher-quality V2 models.

License

MIT.