README.md

# selfhostedmodels

# Evoneural MVP – Local Mesh & Skybox

Localhost MVP for **text → 3D mesh** and **text → 360° skybox** using local models (no hosted APIs).

- **Mesh**: Text → image (Stable Diffusion) → 3D mesh (TripoSR). Output: `.obj` or `.glb`.
- **Skybox**: Text → 2:1 equirectangular image (Stable Diffusion). Optional seamless edge check.

**Default model:** `runwayml/stable-diffusion-v1-5` (no Hugging Face login required; first run downloads ~4GB).

## Prerequisites

- **Python 3.10** (recommended) — [python.org](https://www.python.org/downloads/)
- **NVIDIA GPU** with CUDA (recommended; CPU is slower)
- **Git** (for cloning TripoSR; mesh only)

**No Conda?** Use **venv** (built into Python) — steps below.

## 1. Environment

### Option A: venv + pip (no Conda)

From PowerShell (project folder is `evoneural`):

```powershell
cd D:\project\evoneural
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
```

- **CPU only:** use `pip install torch torchvision` (no `--index-url`).
- **CUDA 12.x:** use `cu121` instead of `cu118`.

### Option B: Conda

```powershell
cd D:\project\evoneural
conda env create -f environment.yml
conda activate evoneural-mvp
```

If you use CPU-only or a different CUDA version, edit `environment.yml` (e.g. remove `pytorch-cuda=11.8` or set `pytorch-cuda=12.1`).

## 2. TripoSR (for mesh)

Mesh generation needs the TripoSR repo and its dependencies.

```powershell
cd D:\project\evoneural
git clone https://github.com/VAST-AI-Research/TripoSR.git TripoSR
pip install -r TripoSR/requirements.txt
```

On Windows, if `torchmcubes` fails, see [TripoSR README](https://github.com/VAST-AI-Research/TripoSR#troubleshooting) (CUDA version match, then reinstall torchmcubes).

## 2b. Stable Diffusion model (Hugging Face)

If you see **"Cannot load model ... model is not cached locally and an error occurred while trying to fetch metadata"**, the app cannot reach Hugging Face. Use one of these:

**Option 1 – Log in (uses cached token)**  
From a terminal with internet:

```powershell
huggingface-cli login
```

Paste a token from [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) (read access is enough). Then run the app again.

**Option 2 – Set token in env**  
Create a token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens), then:

```powershell
$env:HF_TOKEN = "hf_xxxxxxxx"
streamlit run app.py
```

**Option 3 – Download model once, then use offline**  
On a machine that can reach Hugging Face:

```powershell
cd D:\project\evoneural
.venv\Scripts\Activate.ps1
python -m scripts.download_sd_model
```

Then set the path and run the app (no Hugging Face needed):

```powershell
$env:SD_MODEL_PATH = "D:\project\evoneural\weights\stable-diffusion-2-1-base"
streamlit run app.py
```

## How it works

1. **Skybox tab:** You enter a text prompt → the app loads Stable Diffusion (from cache or Hugging Face) → generates a 2:1 image → saves to `outputs/` and shows a download button. Optional “seamless” check compares left/right edges.
2. **Mesh tab:** You enter a prompt (or upload an image) → the app generates an image with SD (if needed) → runs TripoSR on that image → outputs a `.obj` or `.glb` to `outputs/` (requires TripoSR repo cloned in `./TripoSR`).
3. **Model loading:** The app first tries a local folder (`SD_MODEL_PATH` or `weights/stable-diffusion-2-1-base` if complete). If none, it loads `runwayml/stable-diffusion-v1-5` from the Hub (first run downloads the model; later runs use the cache). No token needed unless your network restricts Hugging Face.

## 3. Run the app

From the project root (with venv activated):

```powershell
cd D:\project\evoneural
.venv\Scripts\Activate.ps1
streamlit run app.py
```

Open **http://localhost:8501**.

- **Text → 3D Mesh**: Enter a prompt (or upload an image). First run downloads SD 2.1 and TripoSR weights.
- **Text → Skybox**: Enter a prompt; image is 2:1 (e.g. 1024×512). Use “Run seamless edge check” to compare left/right edges.

Outputs are under `outputs/`. Use the download buttons to save mesh (`.glb`/`.obj`) and skybox (`.png`).

## 4. Performance

- **Skybox**: ~6–8 GB VRAM (SD 2.1, 1024×512, FP16). Use 2048×1024 only if you have enough VRAM.
- **Mesh**: ~6 GB for TripoSR + ~6 GB for SD (text-to-image). Total peak can be ~10–12 GB if both run in same process.

If you run out of VRAM:

- Use 1024×512 for skybox.
- Close other GPU apps.
- Consider quantization (e.g. 8-bit) or CPU offload in diffusers (see [Optimization](#optimization)).

## 5. Optimization (if VRAM is exceeded)

- **Quantization**: Use `load_in_8bit=True` or `load_in_4bit=True` with `bitsandbytes` where supported in diffusers.
- **Model CPU offload**: In diffusers, `pipe.enable_sequential_cpu_offload()` or `pipe.enable_model_cpu_offload()` to move parts to CPU and reduce peak VRAM (slower).
- **Smaller resolution**: 512×512 for text-to-image; 1024×512 for skybox.

## Project layout

```
evoneural/
├── README.md
├── app.py                 # Streamlit UI
├── requirements.txt
├── environment.yml
├── scripts/
│   ├── skybox_generator.py
│   ├── mesh_generator.py
│   ├── text_to_image.py
│   └── check_seamless.py
├── outputs/                # Generated meshes and skybox images
└── TripoSR/                # Clone here (see step 2)
```

## License

See TripoSR and Stable Diffusion model licenses (MIT / Stability). This MVP is for local use and evaluation.