SanskarModi commited on
Commit
b2d52ea
·
1 Parent(s): bb38244

updated readme

Browse files
Files changed (1) hide show
  1. README.md +125 -31
README.md CHANGED
@@ -166,51 +166,145 @@ http://127.0.0.1:7860
166
 
167
  ---
168
 
169
- # Roadmap (LoRA, QLoRA, and Training)
170
 
171
- **Update planned**: full LoRA loading and fine-tuning support.
172
 
173
- Scope includes:
174
 
175
- ### 1. LoRA Runtime Inference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
176
 
177
- * Load LoRA weights into existing UNet
178
- * Adjustable LoRA alpha/scaling
179
- * UI selector for LoRA checkpoints
180
- * Enable mixing multiple LoRAs
 
 
181
 
182
- Implementation plan:
 
183
 
184
- * Attach `lora_attn_procs` to model
185
- * Discover `.safetensors` in `/assets/lora`
186
- * Store LoRA metadata in history
187
- * Persist alpha value and presets
188
 
189
- ### 2. QLoRA Fine-Tuning
 
 
 
 
 
 
190
 
191
- * Train lightweight LoRA modules on GPUs (11GB VRAM OK)
192
- * Use parameter-efficient training
193
- * Merge adapters for export
194
- * Allow user fine-tuning via command line
195
 
196
- Stack:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
197
 
198
- * accelerate
199
- * peft
200
- * bitsandbytes (if GPU available)
201
 
202
- UI tab planned:
203
 
204
- * dataset upload
205
- * config builder
206
- * start training
207
- * track loss, sample outputs
 
208
 
209
- **Why LoRA?**
210
 
211
- * Enables personal styles without training the full model
212
- * Reduces VRAM and compute cost by 50–200×
213
- * Industry-standard for SD customization
214
 
215
  ---
216
 
 
166
 
167
  ---
168
 
169
+ # Roadmap (Focused, High-Impact Features)
170
 
171
+ This project is under active development. The next milestones focus on **practical model customization and multi-model support**, optimized for **CPU-only deployment environments** such as Hugging Face Spaces.
172
 
173
+ The roadmap is intentionally **lean** to maximize value within limited compute constraints.
174
 
175
+ ---
176
+
177
+ ## 1. LoRA Runtime Inference (Core Feature)
178
+
179
+ Add lightweight **Low-Rank Adaptation** support for Stable Diffusion pipelines without modifying base model weights.
180
+
181
+ ### Scope
182
+ - Load external **`.safetensors` LoRA adapters** into UNet
183
+ - Apply LoRA modules dynamically at inference
184
+ - **Alpha (weight) slider** to control influence
185
+ - **UI dropdown** for selecting LoRA adapters
186
+ - **Automatic discovery** of LoRAs under:
187
+ ```
188
+
189
+ src/assets/loras/
190
+
191
+ ```
192
+
193
+ ### Deliverables
194
+ - `lora_loader.py` utility
195
+ - integration into existing `load_pipeline()`
196
+ - UI: LoRA selector + alpha parameter
197
+ - history metadata with:
198
+ - `lora_paths`
199
+ - `lora_weights`
200
+
201
+ ---
202
+
203
+ ## 2. Multi-LoRA Mixing (2 adapters)
204
+
205
+ Support mixing **two LoRA adapters** with independent weights.
206
+
207
+ ### Scope
208
+ - Simple weighted merge at attention processors
209
+ - UI:
210
+ - LoRA A dropdown + alpha
211
+ - LoRA B dropdown + alpha
212
+ - Conflict handling for overlapping layers
213
+
214
+ ### Deliverables
215
+ - `apply_lora_mix()` utility
216
+ - metadata persistence
217
 
218
+ ---
219
+
220
+ ## 3. SDXL-Turbo Pipeline Support
221
+
222
+ Add a **third runtime model**:
223
+ ```
224
 
225
+ stabilityai/stable-diffusion-xl-base
226
+ stabilityai/sdxl-turbo
227
 
228
+ ````
 
 
 
229
 
230
+ ### Scope
231
+ - instantiate SDXL Turbo pipeline
232
+ - auto configure:
233
+ - steps (1-4)
234
+ - CFG (0-1)
235
+ - model selection integrated in UI
236
+ - reproducible metadata
237
 
238
+ ### Notes
239
+ SDXL Turbo is optimized for **fast generation** and works well on constrained environments with reduced steps.
 
 
240
 
241
+ ---
242
+
243
+ ## 4. Enhanced Presets
244
+
245
+ Presets currently define only prompts. Extend them to define **full recommended parameter sets** per use case.
246
+
247
+ ### Scope
248
+ Each preset can define:
249
+ - prompt
250
+ - negative prompt
251
+ - inference steps
252
+ - CFG scale
253
+ - resolution
254
+ - recommended model
255
+ - recommended LoRA (+alpha)
256
+
257
+ ### Example
258
+ ```json
259
+ {
260
+ "preset": "Anime Portrait",
261
+ "prompt": "...",
262
+ "negative": "...",
263
+ "steps": 15,
264
+ "cfg": 6,
265
+ "width": 512,
266
+ "height": 768,
267
+ "model": "SD1.5",
268
+ "lora": {
269
+ "path": "anime_face.safetensors",
270
+ "alpha": 0.8
271
+ }
272
+ }
273
+ ````
274
+
275
+ ---
276
+
277
+ ## 5. Metadata Improvements
278
+
279
+ Enhance metadata tracking for **reproducibility**.
280
+
281
+ ### Added Fields
282
+
283
+ * `model_id`
284
+ * `lora_names`
285
+ * `lora_alphas`
286
+ * `preset_used`
287
+ * `resolution`
288
+ * provenance timestamp
289
+
290
+ This enables exact replication of generated images.
291
+
292
+ ---
293
 
294
+ ## 6. Example LoRA & Training Scripts (No UI)
 
 
295
 
296
+ Provide **self-contained example** to demonstrate training:
297
 
298
+ * a Colab notebook for **LoRA fine-tuning**
299
+ * a small 20-image dataset
300
+ * training duration < 45 minutes on free GPU
301
+ * export `.safetensors` file
302
+ * use it in presets
303
 
304
+ ### Deliverables
305
 
306
+ * `examples/train_lora.ipynb`
307
+ * resulting LoRA stored at `assets/loras/example.safetensors`
 
308
 
309
  ---
310