TECHNICAL_DOCUMENT.md

Guarden — Technical Documentation

This document describes the architecture, data model, and machine-learning components behind Guarden, a Gradio application that helps users identify plants, track a virtual garden, and receive weather-aware watering, care recommendations, health checks...

---

1. High-level architecture

Guarden is a single-process Gradio app (app.py, ~950 lines) backed by a small set of pure-Python modules. There is no database server: each user gets a private, file-based "garden" stored on disk, and three external AI/ML models are called on demand via Hugging Face.

┌─────────────────────────────────────────────────────────────────────┐
│                          Browser (Gradio UI)                         │
│  Location gate → Garden board (drag & drop) → Sidebar (watering /    │
│  forecast / assistant) → Add-plant drawer                            │
└───────────────┬────────────────────────────────────────────────────-┘
                 │ Gradio Blocks events (click / change / .then chains)
┌───────────────▼────────────────────────────────────────────────────-┐
│                              app.py                                   │
│  • Per-user routing (BrowserState user_id → user_data/<uuid>/)       │
│  • Garden CRUD (load/save garden.json, photos, background, links)    │
│  • Board rendering (HTML + SVG overlay + JS drag/drop bridge)         │
│  • Orchestrates calls into modules/* and external APIs                │
└───┬─────────────┬───────────────┬───────────────┬───────────────────┘
    │             │               │               │
    ▼             ▼               ▼               ▼
modules/      modules/        modules/         modules/
classifier.py recommender.py  watering.py       advisor.py
(SigLIP ML     plant.py        weather_utils.py (HF Inference:
 image          (CSV lookup)   (Open-Meteo)       chat LLM +
 classifier)                                      vision LLM)
                                   │
                                   ▼
                          utils/geo.py (Open-Meteo
                                geocoding)

Per-user data layout

user_data/
└── <uuid>/                      # one folder per browser, via gr.BrowserState
    ├── garden.json              # list of plant dicts (see §3)
    ├── background.jpg           # optional custom board background
    └── plant_photos/
        └── <plant_id>.jpg       # uploaded photo for each plant

A user_id (UUID4) is generated on first visit and persisted in the browser's local storage via gr.BrowserState, so the same browser always maps back to the same user_data/<uuid>/ folder — no login/auth required.

---

2. Tech stack

Layer	Technology
UI / app framework	Gradio 6.18 (Blocks API), custom CSS theme (static/style.css), small vanilla-JS bridge for the drag-and-drop board (BOARD_JS in app.py)
Plant genus classification	Fine-tuned SigLIP vision transformer (transformers, local inference, CPU/GPU)
Gardening chat advisor	LLM via Hugging Face Inference Providers (huggingface_hub.InferenceClient)
Photo health diagnostic	Vision-language model via Hugging Face Inference Providers (multimodal chat)
Care metadata	CSV lookup table (data/growth_csv/growth_ds.csv), pandas
Weather & geocoding	Open-Meteo REST APIs (forecast, archive, geocoding) — no API key needed
Persistence	Flat files: garden.json (JSON) + JPEG photos, per user, on local disk
Sprites	Procedurally generated pixel-art PNGs (modules/pixel_art.py, pure PIL, no ML)

Runtime dependencies are pinned in requirements.txt. Training-only dependencies (datasets, accelerate, torchvision) live alongside the inference deps because the classifier's training script ships in the same repo (see §4.1).

---

3. Data model

Each plant in garden.json is a dict with the following fields (collected from how app.py reads/writes them):

{
  "id": "20260610_211240_094893",        // timestamp-based unique id
  "nickname": "Living Room Ficus",       // user-given name
  "photo": "user_data/<uid>/plant_photos/<id>.jpg",
  "genus": "Ficus",                      // predicted by the classifier
  "confidence": 92.4,                    // classifier confidence, %
  "added": "2026-06-10",
  "last_watered": "2026-06-12",          // ISO date or null
  "watering_history": ["2026-06-01", "2026-06-12"], // append-only log
  "rained": false,                       // true if last_watered was inferred from rain
  "watering_frequency_days": "Regular watering",   // raw CSV text
  "sunlight": "full sunlight",
  "soil": "sandy",
  "fertilization_type": "Balanced",
  "notes": "Ficus needs full sunlight. It thrives in sandy soil. ...",
  "position": { "x": 30.0, "y": 40.0 }, // % position on the garden board
  "neighbors": ["<other plant id>"],     // hand-drawn "neighbor" links
  "health": "Healthy — leaves look ..."  // last VLM health diagnosis, if any
}

This structure is the single source of truth: the board, the detail card, the watering table, the advisor and the health diagnostic all read/write this same list of dicts via load_garden(user_id) / save_garden(...).

---

4. Machine-learning components

Guarden uses three distinct AI models, each chosen for a different job: a small fine-tuned vision classifier for genus recognition (fast, local, deterministic), and two Hugging Face Inference-hosted generative models for natural-language and vision-language reasoning (advisor + health check).

4.1 Plant genus classifier (modules/classifier.py)

Task: given a photo of a plant, predict its botanical genus (e.g. Ficus, Aloe, Begonia) out of 289 genus classes.

Model: a fine-tuned google/siglip-base-patch16-224 (SigLIP — a CLIP-style vision transformer, ViT-B/16, 224×224 input, 768-d hidden size) with a SiglipForImageClassification head (289-way softmax). The fine-tuned weights are pushed to a private HF Hub repo (Crocolil/HackatonSmall-storage) and exported as a clean config.json / model.safetensors / preprocessor_config.json bundle (~372 MB) under training/clean_export/.

Training pipeline (training/train_classifier.py):

• Loads a datasets.DatasetDict (train/test split) of labelled plant photos, with one ClassLabel per genus (data/hf_plant_dataset/).
• Builds id2label / label2id from the dataset's ClassLabel feature.
• Data augmentation (train split): RandomResizedCrop(scale=0.8–1.0), RandomHorizontalFlip, ColorJitter(brightness/contrast/saturation=0.1), then ToTensor + SigLIP's own image-mean/std normalization.
• Eval split: deterministic Resize + CenterCrop + normalize.
• Fine-tuned end-to-end with 🤗 Trainer / TrainingArguments:
  • num_train_epochs=3, per_device_*_batch_size=32, lr=5e-5, seed=42
  • bf16=True when CUDA is available
  • eval_strategy="epoch", save_strategy="epoch", load_best_model_at_end=True, metric_for_best_model="accuracy"
  • Metrics: top-1 accuracy and top-5 accuracy (compute_metrics compares argmax / top-5 logits vs. labels).
  • Optional --push-to-hub to publish the checkpoint to a private repo.

Inference (modules/classifier.py):

• CLASSIFIER_MODEL_ID env var points to the Hub repo of the fine-tuned model (loaded lazily, cached as module-level globals).
• classify_plant(image):
  1. AutoImageProcessor resizes/normalizes the uploaded PIL.Image.
  2. AutoModelForImageClassification runs a forward pass (torch.no_grad()).
  3. Softmax over the 289 logits → (genus_name, confidence).
• Called from app.py's add_plants_to_garden() for every uploaded photo; the predicted genus drives everything downstream (care metadata, sprite archetype, advisor context).

4.2 Care recommendation engine (modules/plant.py, modules/recommender.py)

Not a learned model — a deterministic lookup + template layer that turns the classifier's genus output into actionable care info:

• Plant(genus) looks up data/growth_csv/growth_ds.csv (296 genus → care-profile rows, derived from a public plants-growth dataset) for Watering, Sunlight, Soil, Fertilization Type.
• If the genus isn't in the CSV (e.g. a class the classifier knows but the growth table doesn't cover), get_plant_info() falls back to generic defaults ("Water when soil is dry", "indirect sunlight", "well-drained", "No" fertilizer).
• generate_care_notes() assembles a short natural-language note from these fields via string templates (no model call) — shown on the plant detail card under "Notes".

4.3 Watering scheduler (modules/watering.py + modules/weather_utils.py)

Also rule-based, but weather-aware:

• _parse_watering_frequency() maps the CSV's free-text watering instructions (e.g. "Keep soil consistently moist", "Water weekly", "every 10 days") to an integer interval in days, via an exact-match table plus regex fallbacks (DEFAULT_INTERVAL = 4 days if nothing matches).
• should_water(plant, last_watered, date, lat, lon) returns True if:
  • next_watering_date = last_watered + frequency_days has passed, and
  • did_or_will_rain(date, lat, lon, threshold=50%) is False — i.e. it didn't rain in the past (for historical dates) and isn't forecast to rain ≥50% (for today/future), so the app doesn't tell you to water a plant that nature is about to water for you.
• load_garden() also back-fills last_watered from last_rained_date() on every load: if it rained more recently than the recorded watering date, the plant is considered watered by rain (rained: true), avoiding over-watering recommendations after a period of inactivity.
• The sidebar's "Watering today" table is produced by get_watering_recommendations(), which runs should_water() for every plant against the live 7-day forecast.

4.4 AI gardening advisor — chat (modules/advisor.py::ask_about_plant)

Task: free-form Q&A about a specific plant ("Why are the leaves turning yellow?", "Can I plant this next to my tomatoes?").

• Model: ADVISOR_MODEL_ID (default Qwen/Qwen2.5-Coder-3B-Instruct) served via Hugging Face Inference Providers (provider="nscale" by default), through huggingface_hub.InferenceClient.chat_completion.
• Grounding / prompt construction (_build_system_prompt): the system prompt is dynamically built from the plant's care profile (sunlight, soil, watering frequency, fertilization) and its live watering status (computed via _watering_status() from last_watered), so the model knows whether the plant is overdue or recently watered before answering. If the user has drawn "neighbor" links on the board, the linked plants' name/genus are injected too, so the model can reason about companion-planting effects (shared pests, competition for light/water, beneficial pairings).
• The model is instructed to answer in 2–4 sentences, in the same language as the question, and to never recommend toxic/dangerous substances.
• On any InferenceClient error, the function logs [advisor] HF Inference error: ... and returns a friendly fallback message instead of crashing the UI.
• Wired in app.py (ask_plant_advisor) to the "🤖 Ask the assistant" button in the sidebar's Plant assistant panel, which only appears once a plant is selected on the board.

4.5 Photo-based health diagnostic — vision-language (modules/advisor.py::diagnose_plant_health)

Task: given a new photo of the selected plant, assess its health (leaves, stems, soil) and store the verdict on the plant record.

• Model: the same ADVISOR_MODEL_ID / ADVISOR_PROVIDER client as the chat advisor (§4.4), again via InferenceClient.chat_completion — but this time with a multimodal message: the uploaded PIL.Image is re-encoded as JPEG, base64-encoded, and sent as an OpenAI-style content array ({"type": "text", ...} + {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}).
• The prompt asks the model to start its reply with exactly one status word — Healthy, Needs attention, or Sick — followed by a 1–3 sentence explanation and a suggested action.
• app.py::diagnose_selected_plant_health persists the raw model response into plant["health"], which is then surfaced on the plant detail card (🩺 Health: ...) every time the garden is reloaded — so the diagnosis survives page refreshes and is visible alongside the watering history.
• Same defensive error handling as §4.4 ([advisor] HF Inference health-check error: ... + fallback message).

4.6 Procedural pixel-art sprites (modules/pixel_art.py) — not ML, but genus-aware

Worth a short mention because it feels like generative output but is fully deterministic: each genus is mapped to one of 6 hand-authored 16×16 "plant archetype" sprites (cactus, succulent, fern, flower, palm, trailing) and one of 4 pot styles, based on the genus's Growth / Soil / Sunlight values from the same growth_ds.csv (e.g. sandy soil + full sun + slow growth → cactus in a terracotta pot). Genera missing from the CSV get a stable hash-based archetype/pot assignment so the same unknown genus always renders the same sprite. Sprites are rendered once with PIL nearest-neighbour upscaling and cached to static/sprites/<genus>.png.

---

5. External APIs

All weather/geocoding calls go to Open-Meteo (free, no API key):

Function	Endpoint	Used for
utils.geo.city_to_coordinates	geocoding-api.open-meteo.com/v1/search	Turn the user's city into (lat, lon) at the location gate
modules.weather_utils.weather_values	api.open-meteo.com/v1/forecast (16-day daily)	7-day forecast table (conditions, temp, rain %, wind)
modules.weather_utils.did_or_will_rain	forecast (future) or archive-api.open-meteo.com (past)	Decide whether a plant should be watered today / was watered by rain
modules.weather_utils.last_rained_date	archive-api.open-meteo.com/v1/archive (15-day lookback)	Back-fill last_watered on garden load

weather_comment() maps Open-Meteo's numeric WMO weather codes to short emoji + text labels (e.g. 80 → "🌦️ Slight rain showers") shown in the forecast table.

---

6. UI / front-end notes

• Single gr.Blocks app, themed with gr.themes.Soft() plus a large custom stylesheet (static/style.css) that overrides Gradio's CSS variables for a green "Guarden" theme (custom button gradients, card radii, etc.).
• Garden board: plants are rendered as absolutely-positioned <div> sprites inside get_garden_board_html(). A small injected <script> (BOARD_JS) uses pointer events to support:
  • Drag & drop → updates position: {x%, y%} (hidden gr.Number + sync button bridge the JS → Python boundary).
  • Click to select → opens the detail card + action row + assistant panel for that plant.
  • "🔗 Link Neighbours" mode → click two sprites to toggle a neighbors link, drawn as a dashed SVG line between them (re-rendered on every board update, so links follow plants when dragged).
• Sidebar: watering recommendations table, 7-day forecast table (custom CSS turns the Gradio dataframe into card-style rows with a styled header row), and the Plant assistant panel (chat + health diagnostic), which only becomes visible once a plant is selected.
• Per-user custom background: an uploaded image is saved as user_data/<uid>/background.jpg and applied as the board's background-image via inline CSS.

---

7. Deployment

The app is shipped as a Hugging Face Space (sdk: gradio, sdk_version: 6.18.0, entry point app.py, see the README.md front matter). Configuration is entirely via environment variables, with sane defaults baked in so the app runs locally without any secrets:

Env var	Default	Purpose
WEATHER_CITY	"Marseille"	Initial forecast location before the user sets one
CLASSIFIER_MODEL_ID	"your-username/plant-genus-classifier"	HF Hub repo of the fine-tuned SigLIP genus classifier
ADVISOR_MODEL_ID / ADVISOR_PROVIDER	Qwen/Qwen2.5-Coder-3B-Instruct / nscale	Chat advisor + health-diagnostic model and HF Inference provider (shared)
HF_TOKEN	—	Hugging Face token for Inference Providers (advisor + health check)

app.launch(allowed_paths=[...]) whitelists user_data/, static/ and plant_photos/ so per-user photos, sprites and backgrounds can be served back to the browser via Gradio's /gradio_api/file= route.

---