Upload folder using huggingface_hub

README.md

@@ -31,10 +31,11 @@ The trained numeric model uses an AutoScout24 dataset with EUR prices. For the S
 
 1. Enter vehicle details such as make, model, production year, mileage, fuel type, transmission, and power.
 2. Upload front, side, rear, and interior vehicle images.
-3. Receive a base market price prediction.
-4. Receive detected damage categories, image evidence, image-quality warnings, and a transparent damage adjustment.
-5. Receive a recommended listing price range.
-6. Generate an honest used car advertisement text.
+3. Optionally use AI prefill to suggest make, model, year, and body type from the uploaded images.
+4. Receive a base market price prediction.
+5. Receive detected damage categories, image evidence, image-quality warnings, and a transparent damage adjustment.
+6. Receive a recommended listing price range.
+7. Generate an honest used car advertisement text.
 
 ## Repository Structure
 
@@ -58,6 +59,7 @@ Current implemented parts:
 - price preprocessing and feature engineering,
 - numeric price model training and evaluation,
 - OpenAI Vision damage analysis with manual fallback,
+- OpenAI Vision vehicle-field prefill with manual verification,
 - integrated Gradio app for price recommendation, damage adjustment, and listing generation.
 
 ## Local Commands

app/app.py

@@ -12,6 +12,7 @@ from app.integration import IntegratedAnalysis, run_integrated_analysis
 from app.nlp_generator import PromptStrategy
 from app.price_model import PricePrediction, VehicleFeatures
 from app.utils import EUR_TO_CHF_RATE, SWISS_MARKET_FACTOR, eur_to_chf, format_chf
+from app.vehicle_recognition import VehicleRecognition, recognize_vehicle_from_images
 
 
 DAMAGE_CHOICES = ["scratch", "dent", "crack", "lamp broken", "glass shatter", "tire flat"]
@@ -35,6 +36,117 @@ TRANSMISSION_CHOICES = ["Automatic", "Manual", "Semi-automatic", "Unknown"]
 BODY_CHOICES = ["SUV", "Sedan", "Hatchback", "Station Wagon", "Coupe", "Convertible", "Van", "Other"]
 
 
+def map_make_to_ui(make: str | None) -> tuple[str, str]:
+    """Map recognized make to dropdown/custom make fields."""
+    if not make:
+        return "Other", ""
+    for choice in MAKE_CHOICES:
+        if choice.lower() == make.lower():
+            return choice, ""
+    return "Other", make
+
+
+def map_body_type_to_ui(body_type: str | None) -> str:
+    """Map recognized body type to dropdown value."""
+    if not body_type:
+        return "Other"
+    normalized = body_type.strip().lower()
+    aliases = {
+        "estate": "Station Wagon",
+        "wagon": "Station Wagon",
+        "combi": "Station Wagon",
+        "saloon": "Sedan",
+        "convertible": "Convertible",
+        "cabriolet": "Convertible",
+        "hatch": "Hatchback",
+    }
+    if normalized in aliases:
+        return aliases[normalized]
+    for choice in BODY_CHOICES:
+        if choice.lower() == normalized:
+            return choice
+    return "Other"
+
+
+def recognition_html(recognition: VehicleRecognition | None, error: str | None = None) -> str:
+    """Render vehicle-recognition status for the app."""
+    if error:
+        return f"<div class='evidence-warning'><b>AI prefill failed</b><p>{error}</p></div>"
+    if recognition is None:
+        return "<div class='empty-state'>Upload a vehicle image and click AI prefill to suggest vehicle fields.</div>"
+
+    warnings = "".join(f"<li>{warning}</li>" for warning in recognition.warnings)
+    warning_block = f"<ul>{warnings}</ul>" if warnings else "<span>No recognition warnings.</span>"
+    fields = [
+        f"Make: {recognition.make or 'unknown'}",
+        f"Model: {recognition.model or 'unknown'}",
+        f"Year: {recognition.production_year or 'unknown'}",
+        f"Body: {recognition.body_type or 'unknown'}",
+    ]
+    return f"""
+<div class="prefill-panel">
+  <div class="result-label">AI vehicle prefill</div>
+  <p><b>{' | '.join(fields)}</b></p>
+  <p>Confidence: {recognition.confidence:.2f}</p>
+  <p>{recognition.evidence or 'No detailed recognition evidence returned.'}</p>
+  <div class="result-sub">Please verify before valuation.</div>
+  <div class="prefill-warnings">{warning_block}</div>
+</div>
+""".strip()
+
+
+def prefill_vehicle_fields(
+    front_image: str | None,
+    side_image: str | None,
+    rear_image: str | None,
+    interior_image: str | None,
+    current_make: str,
+    current_custom_make: str,
+    current_model: str,
+    current_year: float,
+    current_body_type: str,
+) -> tuple[str, str, str, int, str, str]:
+    """Recognize vehicle identity and update form fields."""
+    image_paths = [
+        ("front", front_image),
+        ("side", side_image),
+        ("rear", rear_image),
+        ("interior", interior_image),
+    ]
+    image_paths = [(view, path) for view, path in image_paths if path]
+    try:
+        recognition = recognize_vehicle_from_images(image_paths)
+    except Exception as exc:  # noqa: BLE001 - user-facing status
+        return (
+            current_make,
+            current_custom_make,
+            current_model,
+            int(current_year),
+            current_body_type,
+            recognition_html(None, str(exc)),
+        )
+
+    make_value, custom_make = map_make_to_ui(recognition.make)
+    if not recognition.make:
+        make_value = current_make
+        custom_make = current_custom_make
+    model_value = recognition.model or current_model
+    year_value = recognition.production_year or int(current_year)
+    body_value = (
+        map_body_type_to_ui(recognition.body_type)
+        if recognition.body_type
+        else current_body_type
+    )
+    return (
+        make_value,
+        custom_make,
+        model_value,
+        year_value,
+        body_value,
+        recognition_html(recognition),
+    )
+
+
 def fallback_price_prediction(features: VehicleFeatures) -> PricePrediction:
     """Provide a conservative fallback if the trained model artifact is missing."""
     age = max(0, 2026 - features.production_year)
@@ -365,6 +477,23 @@ CUSTOM_CSS = """
   margin: 6px 0 0 18px;
   padding: 0;
 }
+.prefill-panel {
+  border: 1px solid #c7d2fe;
+  border-radius: 8px;
+  background: #f8fafc;
+  padding: 12px;
+  margin: 10px 0 0;
+}
+.prefill-panel p {
+  margin: 4px 0;
+  color: #17202a;
+}
+.prefill-warnings ul {
+  margin: 8px 0 0 18px;
+  padding: 0;
+  color: #475569;
+  font-size: 13px;
+}
 .flow-line {
   display: grid;
   grid-template-columns: repeat(3, minmax(0, 1fr));
@@ -489,6 +618,10 @@ def build_demo() -> gr.Blocks:
                             label="Manual fallback damage labels",
                         )
                         gr.HTML(damage_weights_html())
+                    prefill_button = gr.Button("AI prefill vehicle fields", variant="secondary")
+                    prefill_status = gr.HTML(
+                        "<div class='empty-state'>Upload at least one vehicle image to suggest make, model, year, and body type.</div>"
+                    )
 
                 prompt_strategy = gr.Radio(
                     ["Sales-optimized but honest", "Neutral factual"],
@@ -531,6 +664,22 @@ def build_demo() -> gr.Blocks:
             prompt_strategy,
         ]
         outputs = [metrics, flow, vision_evidence, listing, explanation, damage_json, prompt_trace]
+        prefill_button.click(
+            prefill_vehicle_fields,
+            inputs=[
+                front_image,
+                side_image,
+                rear_image,
+                interior_image,
+                make,
+                custom_make,
+                model,
+                production_year,
+                body_type,
+            ],
+            outputs=[make, custom_make, model, production_year, body_type, prefill_status],
+            api_name="prefill_vehicle_fields",
+        )
         run_button.click(predict_listing, inputs=inputs, outputs=outputs, api_name="predict_listing")
     return demo
 

app/vehicle_recognition.py

@@ -0,0 +1,151 @@
+"""Vehicle identification from uploaded images."""
+
+from __future__ import annotations
+
+import base64
+import json
+import mimetypes
+import os
+import re
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+from app.damage_model import DEFAULT_OPENAI_VISION_MODEL
+
+
+@dataclass(frozen=True)
+class VehicleRecognition:
+    """Structured vehicle identification suggestion from image evidence."""
+
+    make: str | None = None
+    model: str | None = None
+    production_year: int | None = None
+    body_type: str | None = None
+    confidence: float = 0.0
+    evidence: str = ""
+    warnings: list[str] = field(default_factory=list)
+
+
+def _image_to_data_url(image_path: Path) -> str:
+    """Encode an image as a data URL for OpenAI vision input."""
+    mime_type = mimetypes.guess_type(str(image_path))[0] or "image/jpeg"
+    encoded = base64.b64encode(image_path.read_bytes()).decode("ascii")
+    return f"data:{mime_type};base64,{encoded}"
+
+
+def _extract_json_object(text: str) -> dict[str, Any]:
+    """Parse a JSON object from model output, even if wrapped in text."""
+    try:
+        parsed = json.loads(text)
+        if isinstance(parsed, dict):
+            return parsed
+    except json.JSONDecodeError:
+        pass
+
+    match = re.search(r"\{.*\}", text, flags=re.DOTALL)
+    if not match:
+        raise ValueError("Vehicle recognition response did not contain a JSON object.")
+    parsed = json.loads(match.group(0))
+    if not isinstance(parsed, dict):
+        raise ValueError("Vehicle recognition response JSON was not an object.")
+    return parsed
+
+
+def _clean_optional_text(value: Any) -> str | None:
+    """Normalize optional text values from a model response."""
+    if value is None:
+        return None
+    text = str(value).strip()
+    if not text or text.lower() in {"unknown", "null", "none", "not visible"}:
+        return None
+    return text
+
+
+def _parse_year(value: Any) -> int | None:
+    """Parse a plausible production-year estimate."""
+    if value is None:
+        return None
+    match = re.search(r"(19|20)\d{2}", str(value))
+    if not match:
+        return None
+    year = int(match.group(0))
+    if 1990 <= year <= 2026:
+        return year
+    return None
+
+
+def recognize_vehicle_from_images(
+    image_paths: list[tuple[str, str | Path]],
+    model: str | None = None,
+) -> VehicleRecognition:
+    """Suggest make, model, approximate year, and body type from vehicle images."""
+    existing_paths = [
+        (view, Path(path))
+        for view, path in image_paths
+        if path and Path(path).exists()
+    ]
+    if not existing_paths:
+        raise FileNotFoundError("Upload at least one vehicle image before using AI prefill.")
+
+    api_key = os.getenv("OPENAI_API_KEY")
+    if not api_key:
+        raise RuntimeError("OPENAI_API_KEY is not configured.")
+
+    from openai import OpenAI
+
+    model_name = model or os.getenv("OPENAI_VISION_MODEL", DEFAULT_OPENAI_VISION_MODEL)
+    prompt = """
+You are a careful used-car identification assistant. Inspect the uploaded car images and infer only visible or strongly likely vehicle facts.
+Return only valid JSON with this schema:
+{
+  "make": "manufacturer or null",
+  "model": "model name or null",
+  "production_year": 2018,
+  "body_type": "SUV|Sedan|Hatchback|Station Wagon|Coupe|Convertible|Van|Other|null",
+  "confidence": 0.0,
+  "evidence": "short visual evidence, such as badges, grille, shape, lights, or interior clues",
+  "warnings": ["uncertainties such as no badge visible, partial image, similar generations, model year approximate"]
+}
+
+Rules:
+- Be conservative. If make/model/year is uncertain, use null or add a warning.
+- Estimate production_year as the most likely visual model year, not registration year.
+- Do not infer mileage, service history, transmission, power, accident history, or fuel type from the image.
+- Prefer broad body_type over exact trim.
+""".strip()
+
+    content: list[dict[str, str]] = [{"type": "input_text", "text": prompt}]
+    for view, path in existing_paths:
+        content.append({"type": "input_text", "text": f"View: {view}"})
+        content.append({"type": "input_image", "image_url": _image_to_data_url(path)})
+
+    client = OpenAI(api_key=api_key)
+    response = client.responses.create(
+        model=model_name,
+        input=[
+            {
+                "role": "user",
+                "content": content,
+            }
+        ],
+    )
+    payload = _extract_json_object(response.output_text)
+    warnings = payload.get("warnings", [])
+    if isinstance(warnings, str):
+        warnings = [warnings]
+
+    confidence = float(payload.get("confidence", 0.0) or 0.0)
+    confidence = max(0.0, min(confidence, 1.0))
+    if confidence < 0.55:
+        warnings.append("Vehicle recognition confidence is low; please verify the fields manually.")
+
+    return VehicleRecognition(
+        make=_clean_optional_text(payload.get("make")),
+        model=_clean_optional_text(payload.get("model")),
+        production_year=_parse_year(payload.get("production_year")),
+        body_type=_clean_optional_text(payload.get("body_type")),
+        confidence=confidence,
+        evidence=str(payload.get("evidence", "")).strip(),
+        warnings=[str(item) for item in warnings if str(item).strip()],
+    )

documentation.md

@@ -35,9 +35,10 @@ The three AI blocks are integrated into one decision flow rather than executed i
 Vehicle attributes -> Numeric ML model -> Base market price
 Vehicle images -> Computer vision model -> Damage categories + evidence + quality warnings + damage severity score
 Base price + damage score + user notes -> NLP component -> Explanation + sales listing
+Vehicle images -> OpenAI Vision prefill -> Suggested make/model/year/body type -> User verification -> Numeric ML model
 ```
 
-The numeric model provides the base price. The computer vision model provides additional information about the vehicle's visible condition. The damage score is then used as a transparent adjustment factor. Finally, the NLP component combines the structured vehicle information, the base price, the damage findings, and the adjusted price into a user-facing explanation and listing text.
+The numeric model provides the base price. The computer vision model provides additional information about the vehicle's visible condition. In addition, an optional OpenAI Vision prefill step can suggest make, model, approximate production year, and body type from uploaded images. The user can verify or edit these fields before valuation. The damage score is then used as a transparent adjustment factor. Finally, the NLP component combines the structured vehicle information, the base price, the damage findings, and the adjusted price into a user-facing explanation and listing text.
 
 ### Planned Application Output
 
@@ -50,7 +51,8 @@ For a given vehicle, the application should return:
 - final recommended listing price range,
 - generated listing text,
 - short explanation of the recommendation,
-- limitations and confidence notes.
+- limitations and confidence notes,
+- optional AI vehicle-field suggestions with confidence and warnings.
 
 ### Scope
 
@@ -330,6 +332,17 @@ The vision prompt asks the model to return structured JSON with visible damage l
 
 The app shows the visual evidence directly in the normal result area, not only in the technical details. If images are blurry, dark, cropped, missing important views, or if model confidence is low, the app adds a warning such as "Damage analysis uncertain". This prevents the interface from overclaiming visual certainty.
 
+### Computer Vision: Vehicle Field Prefill
+
+The optional vehicle-prefill component is implemented in `app/vehicle_recognition.py`. It uses uploaded vehicle images to suggest:
+
+- make,
+- model,
+- approximate production year,
+- body type.
+
+This feature improves usability but does not replace manual input. The app explicitly shows confidence, evidence, and warnings, and the user can correct all fields before the numeric price model is executed. The prefill prompt is conservative: it may return unknown values if the badge, lights, body shape, or model generation are not clear.
+
 Damage-score weights:
 
 | Damage label | Weight |
@@ -378,6 +391,7 @@ Vehicle data -> numeric ML model -> base market price
 Image/manual damage labels -> CV damage component -> evidence + damage score
 Base price + damage score -> adjusted listing price
 Vehicle data + price outputs + damage findings -> NLP -> listing + explanation
+Vehicle images -> optional prefill -> editable vehicle fields -> numeric ML model
 ```
 
 The integration is implemented in `app/integration.py` and exposed through the Gradio interface in `app/app.py`.
@@ -386,6 +400,7 @@ Visible interaction between the blocks:
 
 - the numeric model provides the base market price,
 - the damage component changes the price through the damage score,
+- the optional vehicle-prefill component can populate structured fields from visual evidence,
 - the NLP component receives both the numeric model output and the damage output,
 - the final listing explicitly explains the base price, damage adjustment, visual evidence, Swiss calibration, and recommended listing price.
 
@@ -561,6 +576,7 @@ The deployed Hugging Face Space should be tested with at least three cases:
 | Clean vehicle | normal mileage, clear exterior images, no visible damage | damage score close to 0, no damage discount, listing mentions no visible damage |
 | Minor visible damage | front/side images with scratch or dent | damage label appears, CHF price is adjusted downward, listing mentions the damage transparently |
 | Low-quality images | dark, blurry, cropped, or interior-only images | app shows image-quality warning and avoids overconfident damage claims |
+| Vehicle prefill | clear front/side image with visible badge/body shape | make/model/year/body suggestions appear and remain editable before final analysis |
 
 ### Screenshots
 
@@ -569,7 +585,8 @@ Screenshots for the final report should be stored in `reports/screenshots/`:
 - `01_app_input_multi_image.png`: vehicle form with front, side, rear, and interior upload fields,
 - `02_app_result_clean_vehicle.png`: result with CHF recommendation and no visible damage,
 - `03_app_result_damage_detected.png`: result with damage score, visual evidence, and adjusted price,
-- `04_app_quality_warning.png`: result showing an uncertain image-quality warning.
+- `04_app_quality_warning.png`: result showing an uncertain image-quality warning,
+- `05_app_ai_prefill.png`: AI prefill result with suggested vehicle fields and confidence.
 
 ## 7. Ethical Considerations
 
@@ -579,6 +596,7 @@ To reduce misleading outputs, the app:
 
 - explicitly states that the damage score is heuristic,
 - shows visual evidence and image-quality warnings,
+- treats image-based vehicle recognition as editable suggestions rather than verified facts,
 - keeps manual fallback labels separate from AI image analysis,
 - avoids claiming exact repair costs,
 - generates listing text that mentions visible damage instead of hiding it.
@@ -593,7 +611,8 @@ Current limitations:
 - the Swiss-market factor is transparent but not learned from a Swiss transaction dataset,
 - OpenAI Vision provides image-level visual assessment, not bounding-box localization,
 - image analysis cannot detect hidden mechanical or structural issues,
-- generated listings are deterministic templates and not full LLM-generated marketplace copy.
+- generated listings are deterministic templates and not full LLM-generated marketplace copy,
+- image-based make/model/year recognition can be wrong for visually similar generations or modified vehicles.
 
 Future work:
 

reports/screenshots/README.md

@@ -6,5 +6,6 @@ Use this folder for final documentation screenshots from the deployed Hugging Fa
 - `02_app_result_clean_vehicle.png`: CHF recommendation with no visible damage.
 - `03_app_result_damage_detected.png`: damage score, visual evidence, and adjusted CHF price.
 - `04_app_quality_warning.png`: uncertain image-quality warning for blurry, dark, cropped, or incomplete images.
+- `05_app_ai_prefill.png`: AI vehicle-prefill result with suggested make, model, year, body type, confidence, and warnings.
 
 These screenshots demonstrate the deployed user workflow required in the semester project documentation.