Spaces:
Runtime error
Runtime error
| # Guarden — Technical Documentation | |
| This document describes the architecture, data model, and machine-learning | |
| components behind **Guarden**, a Gradio application that helps | |
| users identify plants, track a virtual garden, and receive weather-aware | |
| watering, care recommendations, health checks... | |
| --- | |
| ## 1. High-level architecture | |
| Guarden is a **single-process Gradio app** (`app.py`, ~950 lines) backed by a | |
| small set of pure-Python modules. There is no database server: each user gets | |
| a private, file-based "garden" stored on disk, and three external AI/ML | |
| models are called on demand via Hugging Face. | |
| ``` | |
| ┌─────────────────────────────────────────────────────────────────────┐ | |
| │ Browser (Gradio UI) │ | |
| │ Location gate → Garden board (drag & drop) → Sidebar (watering / │ | |
| │ forecast / assistant) → Add-plant drawer │ | |
| └───────────────┬────────────────────────────────────────────────────-┘ | |
| │ Gradio Blocks events (click / change / .then chains) | |
| ┌───────────────▼────────────────────────────────────────────────────-┐ | |
| │ app.py │ | |
| │ • Per-user routing (BrowserState user_id → user_data/<uuid>/) │ | |
| │ • Garden CRUD (load/save garden.json, photos, background, links) │ | |
| │ • Board rendering (HTML + SVG overlay + JS drag/drop bridge) │ | |
| │ • Orchestrates calls into modules/* and external APIs │ | |
| └───┬─────────────┬───────────────┬───────────────┬───────────────────┘ | |
| │ │ │ │ | |
| ▼ ▼ ▼ ▼ | |
| modules/ modules/ modules/ modules/ | |
| classifier.py recommender.py watering.py advisor.py | |
| (SigLIP ML plant.py weather_utils.py (HF Inference: | |
| image (CSV lookup) (Open-Meteo) chat LLM + | |
| classifier) vision LLM) | |
| │ | |
| ▼ | |
| utils/geo.py (Open-Meteo | |
| geocoding) | |
| ``` | |
| ### Per-user data layout | |
| ``` | |
| user_data/ | |
| └── <uuid>/ # one folder per browser, via gr.BrowserState | |
| ├── garden.json # list of plant dicts (see §3) | |
| ├── background.jpg # optional custom board background | |
| └── plant_photos/ | |
| └── <plant_id>.jpg # uploaded photo for each plant | |
| ``` | |
| A `user_id` (UUID4) is generated on first visit and persisted in the | |
| browser's local storage via `gr.BrowserState`, so the same browser always | |
| maps back to the same `user_data/<uuid>/` folder — no login/auth required. | |
| --- | |
| ## 2. Tech stack | |
| | Layer | Technology | | |
| |---|---| | |
| | UI / app framework | **Gradio 6.18** (Blocks API), custom CSS theme (`static/style.css`), small vanilla-JS bridge for the drag-and-drop board (`BOARD_JS` in `app.py`) | | |
| | Plant genus classification | Fine-tuned **SigLIP** vision transformer (`transformers`, local inference, CPU/GPU) | | |
| | Gardening chat advisor | LLM via **Hugging Face Inference Providers** (`huggingface_hub.InferenceClient`) | | |
| | Photo health diagnostic | Vision-language model via **Hugging Face Inference Providers** (multimodal chat) | | |
| | Care metadata | CSV lookup table (`data/growth_csv/growth_ds.csv`), `pandas` | | |
| | Weather & geocoding | **Open-Meteo** REST APIs (forecast, archive, geocoding) — no API key needed | | |
| | Persistence | Flat files: `garden.json` (JSON) + JPEG photos, per user, on local disk | | |
| | Sprites | Procedurally generated pixel-art PNGs (`modules/pixel_art.py`, pure PIL, no ML) | | |
| Runtime dependencies are pinned in `requirements.txt`. Training-only | |
| dependencies (`datasets`, `accelerate`, `torchvision`) live alongside the | |
| inference deps because the classifier's training script ships in the same | |
| repo (see §4.1). | |
| --- | |
| ## 3. Data model | |
| Each plant in `garden.json` is a dict with the following fields (collected | |
| from how `app.py` reads/writes them): | |
| ```jsonc | |
| { | |
| "id": "20260610_211240_094893", // timestamp-based unique id | |
| "nickname": "Living Room Ficus", // user-given name | |
| "photo": "user_data/<uid>/plant_photos/<id>.jpg", | |
| "genus": "Ficus", // predicted by the classifier | |
| "confidence": 92.4, // classifier confidence, % | |
| "added": "2026-06-10", | |
| "last_watered": "2026-06-12", // ISO date or null | |
| "watering_history": ["2026-06-01", "2026-06-12"], // append-only log | |
| "rained": false, // true if last_watered was inferred from rain | |
| "watering_frequency_days": "Regular watering", // raw CSV text | |
| "sunlight": "full sunlight", | |
| "soil": "sandy", | |
| "fertilization_type": "Balanced", | |
| "notes": "Ficus needs full sunlight. It thrives in sandy soil. ...", | |
| "position": { "x": 30.0, "y": 40.0 }, // % position on the garden board | |
| "neighbors": ["<other plant id>"], // hand-drawn "neighbor" links | |
| "health": "Healthy — leaves look ..." // last VLM health diagnosis, if any | |
| } | |
| ``` | |
| This structure is the single source of truth: the board, the detail card, | |
| the watering table, the advisor and the health diagnostic all read/write | |
| this same list of dicts via `load_garden(user_id)` / `save_garden(...)`. | |
| --- | |
| ## 4. Machine-learning components | |
| Guarden uses **three** distinct AI models, each chosen for a different job: | |
| a small fine-tuned **vision classifier** for genus recognition (fast, local, | |
| deterministic), and two **Hugging Face Inference**-hosted generative models | |
| for natural-language and vision-language reasoning (advisor + health check). | |
| ### 4.1 Plant genus classifier (`modules/classifier.py`) | |
| **Task**: given a photo of a plant, predict its **botanical genus** (e.g. | |
| `Ficus`, `Aloe`, `Begonia`) out of **289 genus classes**. | |
| **Model**: a fine-tuned [`google/siglip-base-patch16-224`](https://huggingface.co/google/siglip-base-patch16-224) | |
| (SigLIP — a CLIP-style vision transformer, ViT-B/16, 224×224 input, 768-d | |
| hidden size) with a `SiglipForImageClassification` head (289-way softmax). | |
| The fine-tuned weights are pushed to a private HF Hub repo | |
| (`Crocolil/HackatonSmall-storage`) and exported as a clean | |
| `config.json` / `model.safetensors` / `preprocessor_config.json` bundle | |
| (~372 MB) under `training/clean_export/`. | |
| **Training pipeline** (`training/train_classifier.py`): | |
| - Loads a `datasets.DatasetDict` (train/test split) of labelled plant | |
| photos, with one `ClassLabel` per genus (`data/hf_plant_dataset/`). | |
| - Builds `id2label` / `label2id` from the dataset's `ClassLabel` feature. | |
| - Data augmentation (train split): `RandomResizedCrop(scale=0.8–1.0)`, | |
| `RandomHorizontalFlip`, `ColorJitter(brightness/contrast/saturation=0.1)`, | |
| then `ToTensor` + SigLIP's own image-mean/std normalization. | |
| - Eval split: deterministic `Resize` + `CenterCrop` + normalize. | |
| - Fine-tuned end-to-end with 🤗 `Trainer` / `TrainingArguments`: | |
| - `num_train_epochs=3`, `per_device_*_batch_size=32`, `lr=5e-5`, `seed=42` | |
| - `bf16=True` when CUDA is available | |
| - `eval_strategy="epoch"`, `save_strategy="epoch"`, | |
| `load_best_model_at_end=True`, `metric_for_best_model="accuracy"` | |
| - Metrics: **top-1 accuracy** and **top-5 accuracy** | |
| (`compute_metrics` compares `argmax` / top-5 logits vs. labels). | |
| - Optional `--push-to-hub` to publish the checkpoint to a private repo. | |
| **Inference** (`modules/classifier.py`): | |
| - `CLASSIFIER_MODEL_ID` env var points to the Hub repo of the fine-tuned | |
| model (loaded lazily, cached as module-level globals). | |
| - `classify_plant(image)`: | |
| 1. `AutoImageProcessor` resizes/normalizes the uploaded `PIL.Image`. | |
| 2. `AutoModelForImageClassification` runs a forward pass (`torch.no_grad()`). | |
| 3. Softmax over the 289 logits → `(genus_name, confidence)`. | |
| - Called from `app.py`'s `add_plants_to_garden()` for every uploaded photo; | |
| the predicted genus drives everything downstream (care metadata, sprite | |
| archetype, advisor context). | |
| ### 4.2 Care recommendation engine (`modules/plant.py`, `modules/recommender.py`) | |
| Not a learned model — a **deterministic lookup + template** layer that turns | |
| the classifier's genus output into actionable care info: | |
| - `Plant(genus)` looks up `data/growth_csv/growth_ds.csv` (296 genus → | |
| care-profile rows, derived from a public plants-growth dataset) for | |
| `Watering`, `Sunlight`, `Soil`, `Fertilization Type`. | |
| - If the genus isn't in the CSV (e.g. a class the classifier knows but the | |
| growth table doesn't cover), `get_plant_info()` falls back to generic | |
| defaults (`"Water when soil is dry"`, `"indirect sunlight"`, | |
| `"well-drained"`, `"No"` fertilizer). | |
| - `generate_care_notes()` assembles a short natural-language note from these | |
| fields via string templates (no model call) — shown on the plant detail | |
| card under *"Notes"*. | |
| ### 4.3 Watering scheduler (`modules/watering.py` + `modules/weather_utils.py`) | |
| Also rule-based, but **weather-aware**: | |
| - `_parse_watering_frequency()` maps the CSV's free-text watering | |
| instructions (e.g. *"Keep soil consistently moist"*, *"Water weekly"*, | |
| *"every 10 days"*) to an integer interval in days, via an exact-match | |
| table plus regex fallbacks (`DEFAULT_INTERVAL = 4` days if nothing | |
| matches). | |
| - `should_water(plant, last_watered, date, lat, lon)` returns `True` if: | |
| - `next_watering_date = last_watered + frequency_days` has passed, **and** | |
| - `did_or_will_rain(date, lat, lon, threshold=50%)` is `False` — i.e. it | |
| didn't rain in the past (for historical dates) and isn't forecast to | |
| rain ≥50% (for today/future), so the app doesn't tell you to water a | |
| plant that nature is about to water for you. | |
| - `load_garden()` also **back-fills `last_watered`** from `last_rained_date()` | |
| on every load: if it rained more recently than the recorded watering date, | |
| the plant is considered watered by rain (`rained: true`), avoiding | |
| over-watering recommendations after a period of inactivity. | |
| - The sidebar's "Watering today" table is produced by | |
| `get_watering_recommendations()`, which runs `should_water()` for every | |
| plant against the live 7-day forecast. | |
| ### 4.4 AI gardening advisor — chat (`modules/advisor.py::ask_about_plant`) | |
| **Task**: free-form Q&A about a *specific* plant ("Why are the leaves turning | |
| yellow?", "Can I plant this next to my tomatoes?"). | |
| - Model: `ADVISOR_MODEL_ID` (default `Qwen/Qwen2.5-Coder-3B-Instruct`) served | |
| via **Hugging Face Inference Providers** (`provider="nscale"` by default), | |
| through `huggingface_hub.InferenceClient.chat_completion`. | |
| - **Grounding / prompt construction** (`_build_system_prompt`): the system | |
| prompt is dynamically built from the plant's care profile (sunlight, soil, | |
| watering frequency, fertilization) **and** its live watering status | |
| (computed via `_watering_status()` from `last_watered`), so the model knows | |
| whether the plant is overdue or recently watered before answering. If the | |
| user has drawn "neighbor" links on the board, the linked plants' | |
| name/genus are injected too, so the model can reason about | |
| companion-planting effects (shared pests, competition for light/water, | |
| beneficial pairings). | |
| - The model is instructed to answer in **2–4 sentences**, in the same | |
| language as the question, and to never recommend toxic/dangerous | |
| substances. | |
| - On any `InferenceClient` error, the function logs | |
| `[advisor] HF Inference error: ...` and returns a friendly fallback | |
| message instead of crashing the UI. | |
| - Wired in `app.py` (`ask_plant_advisor`) to the "🤖 Ask the assistant" | |
| button in the sidebar's *Plant assistant* panel, which only appears once a | |
| plant is selected on the board. | |
| ### 4.5 Photo-based health diagnostic — vision-language (`modules/advisor.py::diagnose_plant_health`) | |
| **Task**: given a *new* photo of the selected plant, assess its health | |
| (leaves, stems, soil) and store the verdict on the plant record. | |
| - Model: the same `ADVISOR_MODEL_ID` / `ADVISOR_PROVIDER` client as the chat | |
| advisor (§4.4), again via `InferenceClient.chat_completion` — but this time | |
| with a **multimodal** message: the uploaded `PIL.Image` is re-encoded as JPEG, | |
| base64-encoded, and sent as an OpenAI-style content array | |
| (`{"type": "text", ...}` + `{"type": "image_url", "image_url": {"url": | |
| "data:image/jpeg;base64,..."}}`). | |
| - The prompt asks the model to start its reply with exactly one status word | |
| — **`Healthy`**, **`Needs attention`**, or **`Sick`** — followed by a 1–3 | |
| sentence explanation and a suggested action. | |
| - `app.py::diagnose_selected_plant_health` persists the raw model response | |
| into `plant["health"]`, which is then surfaced on the plant detail card | |
| (**🩺 Health:** ...) every time the garden is reloaded — so the diagnosis | |
| survives page refreshes and is visible alongside the watering history. | |
| - Same defensive error handling as §4.4 (`[advisor] HF Inference | |
| health-check error: ...` + fallback message). | |
| ### 4.6 Procedural pixel-art sprites (`modules/pixel_art.py`) — not ML, but genus-aware | |
| Worth a short mention because it *feels* like generative output but is fully | |
| deterministic: each genus is mapped to one of 6 hand-authored 16×16 "plant | |
| archetype" sprites (cactus, succulent, fern, flower, palm, trailing) and one | |
| of 4 pot styles, based on the genus's `Growth` / `Soil` / `Sunlight` values | |
| from the same `growth_ds.csv` (e.g. sandy soil + full sun + slow growth → | |
| cactus in a terracotta pot). Genera missing from the CSV get a **stable | |
| hash-based** archetype/pot assignment so the same unknown genus always | |
| renders the same sprite. Sprites are rendered once with PIL nearest-neighbour | |
| upscaling and cached to `static/sprites/<genus>.png`. | |
| --- | |
| ## 5. External APIs | |
| All weather/geocoding calls go to **Open-Meteo** (free, no API key): | |
| | Function | Endpoint | Used for | | |
| |---|---|---| | |
| | `utils.geo.city_to_coordinates` | `geocoding-api.open-meteo.com/v1/search` | Turn the user's city into `(lat, lon)` at the location gate | | |
| | `modules.weather_utils.weather_values` | `api.open-meteo.com/v1/forecast` (16-day daily) | 7-day forecast table (conditions, temp, rain %, wind) | | |
| | `modules.weather_utils.did_or_will_rain` | forecast (future) or `archive-api.open-meteo.com` (past) | Decide whether a plant should be watered today / was watered by rain | | |
| | `modules.weather_utils.last_rained_date` | `archive-api.open-meteo.com/v1/archive` (15-day lookback) | Back-fill `last_watered` on garden load | | |
| `weather_comment()` maps Open-Meteo's numeric WMO weather codes to short | |
| emoji + text labels (e.g. `80` → "🌦️ Slight rain showers") shown in the | |
| forecast table. | |
| --- | |
| ## 6. UI / front-end notes | |
| - Single `gr.Blocks` app, themed with `gr.themes.Soft()` plus a large custom | |
| stylesheet (`static/style.css`) that overrides Gradio's CSS variables for | |
| a green "Guarden" theme (custom button gradients, card radii, etc.). | |
| - **Garden board**: plants are rendered as absolutely-positioned `<div>` | |
| sprites inside `get_garden_board_html()`. A small injected `<script>` | |
| (`BOARD_JS`) uses pointer events to support: | |
| - **Drag & drop** → updates `position: {x%, y%}` (hidden `gr.Number` + | |
| sync button bridge the JS → Python boundary). | |
| - **Click to select** → opens the detail card + action row + assistant | |
| panel for that plant. | |
| - **"🔗 Link Neighbours" mode** → click two sprites to toggle a | |
| `neighbors` link, drawn as a dashed SVG line between them (re-rendered | |
| on every board update, so links follow plants when dragged). | |
| - **Sidebar**: watering recommendations table, 7-day forecast table (custom | |
| CSS turns the Gradio dataframe into card-style rows with a styled header | |
| row), and the *Plant assistant* panel (chat + health diagnostic), which | |
| only becomes visible once a plant is selected. | |
| - **Per-user custom background**: an uploaded image is saved as | |
| `user_data/<uid>/background.jpg` and applied as the board's | |
| `background-image` via inline CSS. | |
| --- | |
| ## 7. Deployment | |
| The app is shipped as a **Hugging Face Space** (`sdk: gradio`, | |
| `sdk_version: 6.18.0`, entry point `app.py`, see the `README.md` front | |
| matter). Configuration is entirely via environment variables, with sane | |
| defaults baked in so the app runs locally without any secrets: | |
| | Env var | Default | Purpose | | |
| |---|---|---| | |
| | `WEATHER_CITY` | `"Marseille"` | Initial forecast location before the user sets one | | |
| | `CLASSIFIER_MODEL_ID` | `"your-username/plant-genus-classifier"` | HF Hub repo of the fine-tuned SigLIP genus classifier | | |
| | `ADVISOR_MODEL_ID` / `ADVISOR_PROVIDER` | `Qwen/Qwen2.5-Coder-3B-Instruct` / `nscale` | Chat advisor + health-diagnostic model and HF Inference provider (shared) | | |
| | `HF_TOKEN` | — | Hugging Face token for Inference Providers (advisor + health check) | | |
| `app.launch(allowed_paths=[...])` whitelists `user_data/`, `static/` and | |
| `plant_photos/` so per-user photos, sprites and backgrounds can be served | |
| back to the browser via Gradio's `/gradio_api/file=` route. | |
| --- | |