|
|
--- |
|
|
license: apache-2.0 |
|
|
base_model: runwayml/stable-diffusion-v1-5 |
|
|
tags: |
|
|
- stable-diffusion |
|
|
- diffusers |
|
|
- lora |
|
|
- scientific-machine-learning |
|
|
- physics-simulation |
|
|
- visual-reasoning |
|
|
--- |
|
|
|
|
|
# VR-LoRA: N-Body Physics Simulator (SD-v1.5) |
|
|
|
|
|
This repository contains a Visual Reasoning LoRA (VR-LoRA) fine-tuned on `runwayml/stable-diffusion-v1-5` to predict the temporal evolution of a 3-body gravitational system. |
|
|
|
|
|
This model is the primary output of the paper: **"Visual Reasoning Transfer: Leveraging Pretrained Visual Models for Physical and Temporal Prediction"** (link to paper coming soon). |
|
|
|
|
|
## Model Description |
|
|
|
|
|
The model does not generate images. Instead, it acts as a dynamics engine in latent space. When given a latent representation of a physical state (encoded as a "spatial field image"), it predicts the latent representation of the next physical state. |
|
|
|
|
|
This LoRA was trained for 15,000 steps on a synthetic dataset of 10,000 N-body trajectories. |
|
|
|
|
|
* **Research Project Repository:** [https://github.com/sandner-art/SC-Visual-Reasoning](https://github.com/sandner-art/SC-Visual-Reasoning) |
|
|
* **Training Dataset:** [huggingface.co/datasets/sandner/n-body-trajectories-for-vrlora](https://huggingface.co/datasets/sandner/n-body-trajectories-for-vrlora) (Replace with your link) |
|
|
|
|
|
## How to Use |
|
|
|
|
|
This LoRA is designed to be used with the evaluation script found in the main research repository. It is not intended for standard text-to-image generation. |
|
|
|
|
|
```python |
|
|
# See the evaluate_vr_lora.py script in the main GitHub repo for a full example. |
|
|
from diffusers import StableDiffusionPipeline |
|
|
|
|
|
base_model = "runwayml/stable-diffusion-v1-5" |
|
|
lora_model = "sandner/vr-lora-physics-sd15" # Replace with your repo name |
|
|
|
|
|
pipeline = StableDiffusionPipeline.from_pretrained(base_model) |
|
|
pipeline.load_lora_weights(lora_model) |
|
|
pipeline.to("cuda") |
|
|
|
|
|
# Now the `pipeline.unet` component is ready for physics simulation. |
|
|
unet = pipeline.unet |
|
|
# ... |
|
|
``` |
|
|
|
|
|
## Training Procedure |
|
|
|
|
|
The model was trained using the `train_vr_lora.py` script from the project repository. Key hyperparameters: |
|
|
|
|
|
* **Learning Rate:** 1e-4 |
|
|
* **Batch Size:** 32 |
|
|
* **Max Steps:** 15,000 |
|
|
* **Optimizer:** AdamW |
|
|
* **Scheduler:** Cosine |
|
|
|
|
|
## Citing this Work |
|
|
|
|
|
If you use this model in your research, please cite our paper (BibTeX entry coming soon). |
|
|
|
