MINDI 1.5 Vision-Coder — Complete Project Context
Last updated: May 2, 2026 (Session 5) Purpose: This file contains ALL context needed to continue development with any AI assistant. It covers architecture decisions, errors encountered, fixes applied, training state, frontend state, and exact next steps.
1. PROJECT OVERVIEW
MINDI 1.5 Vision-Coder is a multimodal AI model that generates frontend code (HTML/CSS/JS, Next.js, Tailwind) from UI screenshots and text prompts. It combines:
- Qwen/Qwen2.5-Coder-7B-Instruct — 7.62B param base LLM (Apache 2.0)
- CLIP ViT-L/14 — Frozen vision encoder for UI screenshot understanding
- LoRA adapters — Efficient fine-tuning (r=64, alpha=128)
- Vision-Language Fusion — Prepend visual tokens to text embeddings
- 22 MINDI Special Tokens — Structured agentic reasoning (think, code, critique, fix, etc.)
- 3-Phase Training Strategy — Progressive training on MI300X 192GB
Repos:
- GitHub:
https://github.com/Faaz345/MINDI-1.5-Vision-Coder.git(branch:master) - HuggingFace Model:
Mindigenous/MINDI-1.5-Vision-Coder(private, push asmaster:main) - HuggingFace Dataset:
Mindigenous/MINDI-1.5-training-data(private) - HuggingFace Space:
Mindigenous/mindi-chat— live Gradio 5.x Space (ZeroGPU) - HF Token: Set as
HF_TOKENenvironment variable (stored separately, not in repo)
2. TRAINING STATUS — COMPLETE ✅
All 3 phases of MINDI 1.5 Vision-Coder training are COMPLETE:
| Phase | Steps | Status | Platform |
|---|---|---|---|
| Phase 1 (LoRA) | 5,000 | ✅ Complete | DigitalOcean MI300X |
| Phase 2 (Vision Bridge) | 2,500 | ✅ Complete | DigitalOcean MI300X |
| Phase 3 (Joint) 0→1500 | 1,500 | ✅ Complete | DigitalOcean MI300X |
| Phase 3 (Joint) 1500→2500 | 1,000 | ✅ Complete | Modal A100-40GB |
Final loss: 0.25–0.40 range
VRAM: 17.2 GB on A100-40GB
All checkpoints: Uploaded to checkpoints/ in HF model repo
HuggingFace Checkpoints (Mindigenous/MINDI-1.5-Vision-Coder)
- Phase 1: 16 checkpoints (step250 → step5000)
- Phase 2: 10 checkpoints (step250 → step2500)
- Phase 3:
phase3_all_step500,step1000,step1500,step2000,phase3_all_step2500_final,phase3_final
3. LIVE API — HuggingFace SPACE
Space URL: https://mindigenous-mindi-chat.hf.space
Space ID: Mindigenous/mindi-chat
Framework: Gradio 5.23.0 (ZeroGPU)
Protocol: SSE v3 — two-step: POST to submit → GET to stream result
API Call Pattern (Gradio 5.x SSE v3)
// Step 1: Submit job
POST https://mindigenous-mindi-chat.hf.space/gradio_api/call/chat_fn
Headers: { "Content-Type": "application/json", "Authorization": "Bearer hf_..." }
Body: { "data": [prompt, imageArg, temperature, maxTokens, historyJson] }
Response: { "event_id": "..." }
// Step 2: Stream result
GET https://mindigenous-mindi-chat.hf.space/gradio_api/call/chat_fn/{event_id}
Parse SSE: find "event: complete" → next line "data: [...]"
Parse data[0] as JSON: { response: "...", sections: {} }
ZeroGPU Quota
- Anonymous users: Very low quota (hits "GPU task aborted" error quickly)
- Authenticated users (HF token): ~8× higher quota
- Quota errors throw as exceptions with message containing "GPU task aborted" or "zerogpu"
- Fix: Always send
Authorization: Bearer <HF_TOKEN>header
Gradio Function Signature
# hf_space/app.py — chat_fn
def chat_fn(prompt: str, image: dict|None, temperature: float, max_tokens: int, history_json: str) -> str:
# Returns JSON string: {"response": "...", "sections": {...}}
4. FRONTEND — NEW VITE + REACT WEBSITE BUILDER ⭐ (Session 5 Work)
What Was Built (May 2, 2026)
The old vanilla HTML/CSS/JS chat frontend was completely replaced with a professional 3-panel website builder IDE (similar to Bolt.new / v0.dev), built with Vite + React.
The old frontend is backed up in: frontend/_legacy/
How to Run
cd "d:\Desktop 31st Jan 2026\MINDI 1.5 vision-coder\frontend"
npm install # only first time
npm run dev # starts at http://localhost:5173
New Frontend Structure
frontend/
├── index.html # Shell with Google Fonts (Inter + JetBrains Mono)
├── package.json # Vite 8.x + React 19 + prismjs + lucide-react
├── vite.config.js # Vite config
├── _legacy/ # Old vanilla JS chat frontend (backed up)
└── src/
├── main.jsx # React entry point
├── index.css # Design system (CSS tokens, reset, animations)
├── App.jsx # Main app — all state management + generation flow
├── App.css # All layout + component styles (3-panel IDE)
├── components/
│ ├── Sidebar.jsx # File tree + Agent Progress + status indicator
│ ├── Editor.jsx # Code editor with line-by-line animation + tabs
│ ├── Preview.jsx # Always-visible iframe preview + Console panel
│ ├── PromptBar.jsx # Bottom prompt input (auto-resize, send/stop)
│ ├── PlanModal.jsx # Clarifying questions (tech stack, design style)
│ ├── SettingsModal.jsx # API URL, HF token, temperature, max tokens
│ └── Toasts.jsx # Toast notifications
└── services/
├── api.js # Gradio SSE v3 integration + auth + demo fallback
├── promptEnhancer.js # Analyzes prompt → asks questions → structured prompt
└── fileParser.js # Extracts files from model response markdown
Layout
┌──────────────┬─────────────────────────┬──────────────────┐
│ SIDEBAR │ CODE EDITOR │ LIVE PREVIEW │
│ (260px) │ (flex: 1) │ (420px) │
│ │ │ │
│ MINDI 1.5 │ 🌐 index.html │ ● Preview │
│ brand │ 1 <!DOCTYPE html> │ [Rendered HTML] │
│ │ 2 <html lang="en"> │ │
│ FILES (1) │ 3 <head> │ │
│ 🌐 index. │ ... │ CONSOLE │
│ html │ │ > Page rendered │
│ │ │ │
│ AGENT │ │ │
│ PROGRESS │ │ │
│ ✅ Enhancing│ │ │
│ ✅ Generating│ │ │
│ ✅ Complete │ │ │
│ │ │ │
│ ● MINDI · │ │ │
│ Connected │ │ │
├──────────────┴─────────────────────────┴──────────────────┤
│ [Describe what you want to build...] [Send] │
│ MINDI 1.5 Vision-Coder Shift+Enter new line │
└────────────────────────────────────────────────────────────┘
Key Features
Plan Modal — When user submits prompt without specifying tech stack or theme, a "Configure Your Project" modal appears with:
- Tech stack: HTML+CSS+JS / React / Next.js / Vue
- Design style: Dark / Light / Gradient / Minimal
- "Skip & Generate" and "Generate ⚡" buttons
Prompt Enhancer (
src/services/promptEnhancer.js) — Transforms raw input into structured prompts with design requirements, responsiveness rules, font choices, no-placeholder rules.Code Animation — Lines appear one by one at 15ms intervals with
line-appearCSS animation as code generates.File Tree — Files parsed from model response appear in sidebar with fade-in animation. Click to switch active file in editor.
Live Preview — Always-visible iframe on right renders HTML output. "Open in new tab" and "Copy HTML" buttons.
Demo Fallback — When API quota exceeded or any error occurs, pre-built demo responses for common prompts (landing page, dashboard) render automatically. No white screen.
Settings — Click the MINDI logo (top-left) to open Settings: configure API URL, HF Token, Temperature, Max Tokens.
Error Handling in api.js
// Two separate detection mechanisms:
isQuotaError(result) // Response-level: checks result.response + result.sections.error
isQuotaException(errMsg) // Exception-level: checks thrown error message
// Both match: zerogpu | gpu quota | gpu task aborted | task aborted | unlogged user
When quota error detected → immediately falls back to generateDemo(prompt) which returns pre-built HTML.
Demo Responses Available
/landing|hero|page|website/i→ Lumina landing page (Tailwind, gradient, features section)/dashboard|chart|analytics|admin/i→ Pulsegrid dashboard (sidebar, stat cards, bar chart)- Default → Simple MINDI hello card
Settings Persistence
Saved in localStorage under key mindi.builder.v1:
{
"apiUrl": "https://mindigenous-mindi-chat.hf.space",
"hfToken": "hf_...",
"temperature": 0.7,
"maxTokens": 2048
}
5. DIRECTORY STRUCTURE (Full Project)
MINDI-1.5-Vision-Coder/
├── src/ # Model source code
│ ├── model/
│ │ ├── architecture.py # Qwen2.5-Coder + LoRA wrapper (NOT nn.Module)
│ │ ├── mindi_model.py # MINDI15 main class (nn.Module)
│ │ ├── vision_encoder.py # CLIP ViT-L/14 (frozen) + trainable projection
│ │ ├── fusion_layer.py # VisionLanguageFusion with text_gate
│ │ └── __init__.py
│ ├── training/
│ │ ├── mindi_trainer.py # MINDITrainer: 3-phase loop, streaming data
│ │ ├── data_pipeline.py # Data processing pipeline
│ │ └── __init__.py
│ └── ...
├── scripts/
│ ├── train.py # Master training launcher
│ ├── download_websight.py
│ ├── upload_websight_images.py
│ └── gpu_diagnostic.py
├── hf_space/
│ ├── app.py # Gradio Space — live at Mindigenous/mindi-chat
│ └── requirements.txt
├── frontend/ # ⭐ NEW: Vite + React website builder
│ ├── index.html
│ ├── package.json
│ ├── _legacy/ # Old vanilla JS chat (backup)
│ └── src/ # (see Section 4 above)
├── api/ # FastAPI endpoints (future)
├── modal_api.py # Modal.com A100 API server
├── modal_train.py # Modal.com training script
├── data/ # Local training data
├── configs/ # Training configs
├── context.md # ← THIS FILE
└── ...
6. ARCHITECTURE DETAILS
6.1 Model Components
| Component | Class | File | Params | Trainable |
|---|---|---|---|---|
| Base LLM | MINDIArchitecture |
architecture.py |
7.62B | No (frozen) |
| LoRA | via PEFT | architecture.py |
161.5M | Yes |
| CLIP Vision | VisionEncoder |
vision_encoder.py |
304M | 4.2M (projection only) |
| Fusion | VisionLanguageFusion |
fusion_layer.py |
16.8M | Yes |
| Total | MINDI15 |
mindi_model.py |
8.1B | 182.5M (2.25%) |
6.2 CRITICAL Architecture Notes
MINDIArchitectureis NOT annn.Module— it's a plain Python wrapper. The actual trainable PeftModel is accessed viaself.architecture.get_model()and registered asself.llminMINDI15.__init__().self.llm = self.architecture.get_model()— Required somodel.parameters()finds LoRA params.Fusion layer has
text_gate— Learnable scalar (init=0) for gradient flow during text-only batches.
6.3 MINDI Special Tokens (22 total, 11 pairs)
<|think_start|> / <|think_end|> — Internal reasoning
<|code_start|> / <|code_end|> — Generated code blocks
<|file_start|> / <|file_end|> — File references
<|critique_start|> / <|critique_end|> — Self-critique
<|suggest_start|> / <|suggest_end|> — Suggestions
<|search_start|> / <|search_end|> — Search context
<|error_start|> / <|error_end|> — Error messages
<|fix_start|> / <|fix_end|> — Fix attempts
<|vision_start|> / <|vision_end|> — Vision input markers
<|sandbox_start|> / <|sandbox_end|> — Sandbox execution
<|context_start|> / <|context_end|> — Context block
7. HF SPACE — app.py KEY DETAILS
File: hf_space/app.py
System Prompt (no identity hallucination fix)
The system prompt explicitly states: "You are MINDI 1.5 Vision-Coder, created by Mindigenous. You are NOT GPT-4, Claude, or any other AI..."
chat_fn Signature
@spaces.GPU(duration=60)
def chat_fn(prompt, image, temperature, max_tokens, history_json):
# history_json is a JSON string of [{"role": ..., "content": ...}, ...]
# Returns: JSON string {"response": "...", "sections": {...}}
Gradio Interface
gr.Interface(
fn=chat_fn,
inputs=[
gr.Textbox(label="Prompt"),
gr.Image(type="filepath", label="Image"),
gr.Slider(0, 2, value=0.7, label="Temperature"),
gr.Slider(128, 4096, value=2048, label="Max Tokens"),
gr.Textbox(label="History JSON", visible=False),
],
outputs=gr.Textbox(label="Response"),
api_name="chat_fn"
)
8. KNOWN ERRORS & FIXES HISTORY
Training Errors (all fixed ✅)
| # | Error | Fix |
|---|---|---|
| 6.1 | GPU hang — HSA_OVERRIDE_GFX_VERSION | Do NOT set this var on ROCm 7.0 |
| 6.2 | No trainable params in optimizer | self.llm = self.architecture.get_model() |
| 6.3 | extra_special_tokens format error | Changed from list to dict in tokenizer_config.json |
| 6.4 | Phase 2 gradient flow crash | Added text_gate residual in VisionLanguageFusion |
| 6.5 | Git LFS push failures | .gitattributes + git lfs migrate import |
| 6.6 | HF auth for MI300X clone | Use token as both username+password in git URL |
| 6.7 | GPU hang after heavy I/O | PCI reset: echo 1 > /sys/bus/pci/devices/0000:83:00.0/reset |
| 6.8 | HF upload limits (10K/dir, 25K/commit) | Reorganized images into 6 subdirs |
| 6.9 | snapshot_download HTTP 429 | Use git clone instead |
| 6.10 | Bash history expansion !' |
Use multi-line python or single-quoted strings |
| 6.11 | Data dir already exists on clone | rm -rf data before cloning dataset repo |
Frontend API Errors (all fixed ✅)
| # | Error | Fix |
|---|---|---|
| 6.12 | handleSend ReferenceError in old app.js |
let activeSend = send pattern (now in _legacy) |
| 6.13 | Gradio 3.x → 5.x API mismatch (404 on /api/predict) | Rewrote to SSE v3 two-step flow |
| 6.14 | Health check misdetects Space as offline | Use fetch(base, {mode:'no-cors'}) for HF Spaces |
| 6.15 | GPU quota blocks demo — no fallback | isQuotaError() + isQuotaException() → auto demo |
| 6.16 | handlePlanSubmit catch had no demo fallback | Added demo fallback to all catch blocks in App.jsx |
9. SESSION HISTORY
| Session | Date | Key Work |
|---|---|---|
| 1 | April 15, 2026 | Phase 1 dry run. GPU hang resolved. |
| 2 | April 16, 2026 | Phase 1 training 0→4250. WebSight data uploaded. |
| 3 | April 19–28, 2026 | Phase 1→2→3 complete. Model deployed to HF Space. |
| 4 | April 30, 2026 | Fixed Gradio API protocol. HF token auth. ZeroGPU quota handling. Agent scaffolded. |
| 5 | May 2, 2026 | Rebuilt frontend as Vite+React 3-panel IDE. Prompt enhancer, plan modal, code animation, live preview, file tree, demo fallback. |
10. WHAT WORKS ✅
- Model training — All 3 phases complete, checkpoints on HF
- HF Space — Live at
Mindigenous/mindi-chat, Gradio 5.x SSE v3 - New Frontend (Vite+React) —
http://localhost:5173- 3-panel IDE (Sidebar | Editor | Preview)
- Plan Modal (tech stack + design style questions)
- Prompt Enhancer (raw input → structured prompt)
- Code animation (line-by-line fade-in)
- File tree (real-time population during generation)
- Live preview (always-visible iframe)
- Demo fallback (landing page + dashboard demos)
- Settings modal (API URL, HF token, temperature)
- ZeroGPU quota detection + auto-fallback
- Build —
npm run build→ 222KB JS (70KB gzip), 3.25s
11. WHAT REMAINS ❌
High Priority
- Add HF token to Settings — Without token, demo fallback always used. Real MINDI output requires
hf_...token in Settings modal. - Make suggestion pills clickable — "Landing Page", "Dashboard" etc. chips on welcome screen should trigger generation when clicked.
- Syntax highlighting — Add Prism.js token coloring to the code editor.
Medium Priority
- Vision loop — Feed preview screenshots back to MINDI for automated visual QA (captureScreenshot → base64 → callMINDI).
- Multi-file support — Model generates single-file HTML currently. Add prompt instruction for
// filename:markers to split into HTML/CSS/JS. - Download project button — Let user download generated files as a ZIP.
Low Priority
- WebContainer SDK — For projects that need Node.js execution (Next.js, npm packages).
- Fine-tuning for multi-file output — Train on structured output format with
// filename:markers. - Deploy frontend — Host on Vercel or GitHub Pages (free).
12. NEXT SESSION CHECKLIST
When starting a new AI assistant session:
- Read this file first (most important)
- Run frontend:
cd "d:\Desktop 31st Jan 2026\MINDI 1.5 vision-coder\frontend" npm run dev # Opens at http://localhost:5173 - Add HF token in Settings (click MINDI logo → Settings → paste
hf_...token) - Test with real MINDI model — type "landing page", skip plan modal, verify real response comes back
- Continue from "What Remains" section above — start with suggestion chips or syntax highlighting
13. COMMANDS REFERENCE
Frontend (Windows PowerShell)
# Run dev server
cd "d:\Desktop 31st Jan 2026\MINDI 1.5 vision-coder\frontend"
npm run dev # http://localhost:5173
# Build for production
npm run build # dist/ folder
# Check build
npx vite build 2>&1 | Select-Object -Last 10
Git
git add -A
git commit -m "..."
git push origin master # GitHub
git push hf master:main # HuggingFace
Local (Windows, PowerShell, in venv)
& ".\venv\Scripts\Activate.ps1"
$env:HF_TOKEN="<your-hf-token>"
python scripts/download_websight.py --num_train 50000 --num_val 2500
python scripts/upload_websight_images.py
MI300X (if spinning up again)
export HF_TOKEN=<your-hf-token>
export PYTORCH_ROCM_ARCH=gfx942
export TOKENIZERS_PARALLELISM=false
# DO NOT SET: HSA_OVERRIDE_GFX_VERSION
# GPU test
python3 -c "import torch; print('GPU:', torch.cuda.get_device_name(0)); x=torch.randn(100,device='cuda'); print('OK:', x.sum().item())"
# Full training
nohup python3 scripts/train.py --no_wandb > /workspace/training.log 2>&1 &
14. DESIGN SYSTEM (Frontend)
CSS variables defined in src/index.css:
--bg-0: #07070c; /* Deepest background */
--bg-1: #0a0a12;
--panel: #111120; /* Sidebar, modals */
--border: rgba(255,255,255,.06);
--text: #ececf1;
--text-2: #b4b4c4;
--text-mute: #7a7a8c;
--purple: #7c3aed;
--purple-light: #a78bfa;
--blue: #2563eb;
--grad: linear-gradient(135deg, #7c3aed 0%, #2563eb 100%);
--sans: 'Inter', ...;
--mono: 'JetBrains Mono', ...;
--sidebar-w: 260px;
Key animations: fadeIn, line-appear, float, pulse, spin, pop-in, toast-in
15. MODEL QUALITY NOTES
MINDI 1.5 is a 7B model with ~10K training steps. Known characteristics:
| Issue | Status | Mitigation |
|---|---|---|
| Identity hallucination ("I am GPT-4") | ✅ Fixed via system prompt | Strong MINDI identity in hf_space/app.py |
| Basic/simple HTML output | ⚠️ Expected for 7B | Prompt enhancer adds design requirements |
| Weak image understanding | ⚠️ Only 2.5K vision steps | Prompt still works for text-only generation |
| No multi-file output | ⚠️ Not trained on it | Single complete file works fine |
The prompt enhancer compensates for most quality issues by structuring prompts with explicit design requirements (fonts, colors, responsiveness, no-placeholders rule, complete code requirement).
Updated May 2, 2026 — Session 5: Rebuilt frontend as Vite+React 3-panel website builder IDE. Previous sessions: April 15–30, 2026 — Model training (3 phases), HF Space deployment, API fixes.