made it faster and added color picker

app/SAMPLE_DATA_README.md

@@ -1,141 +0,0 @@
-# Sample Housing Dataset - Comprehensive Real Estate Analytics
-
-## Overview
-This enhanced sample dataset contains **500 realistic property listings** with **33 features** designed to enable comprehensive real estate market analysis and insightful dashboard creation.
-
-## Dataset Features
-
-### Basic Property Information (9 columns)
-- **price**: Property sale price ($361K - $5.08M)
-- **property_type**: Single Family, Condo, Townhouse, or Multi-Family
-- **neighborhood**: Downtown, Suburbs, Waterfront, Historic District, or Business District
-- **bedrooms**: Number of bedrooms (1-6)
-- **bathrooms**: Number of bathrooms (1.0-4.5)
-- **sqft_living**: Interior living space in square feet (800-5,500)
-- **sqft_lot**: Lot size in square feet (2,000-20,000)
-- **floors**: Number of floors (1.0-3.0)
-- **year_built**: Year the property was built (1950-2023)
-
-### Property Characteristics (7 columns)
-- **year_renovated**: Year of last major renovation (0 if never renovated)
-- **property_age**: Age of the property in years
-- **waterfront**: Boolean - waterfront property (12% have this premium feature)
-- **view_quality**: View rating from 0-4 (higher is better)
-- **condition**: Overall condition rating 1-5
-- **grade**: Construction quality grade 3-12 (7-8 is average)
-- **price_segment**: Categorized as Budget, Mid-Range, Premium, or Luxury
-
-### Amenities & Features (4 columns)
-- **parking_spaces**: Number of parking spaces (0-4)
-- **has_pool**: Boolean - property has a swimming pool (15%)
-- **has_fireplace**: Boolean - property has fireplace(s) (35%)
-- **has_basement**: Boolean - property has a basement (45%)
-
-### Neighborhood Quality (3 columns)
-- **school_rating**: School district rating (3-10, higher is better)
-- **crime_index**: Neighborhood crime index (15-85, lower is better)
-- **walkability_score**: Walkability score (30-100, higher is better)
-
-### Market Metrics (3 columns)
-- **price_per_sqft**: Price per square foot (calculated)
-- **days_on_market**: Days the property was listed (1-180)
-- **hoa_fees**: Monthly HOA fees ($0-$800, ~40% of properties)
-
-### Temporal Data (4 columns)
-- **sale_date**: Date of sale (2022-01-01 to 2024-01-01)
-- **sale_month**: Month of sale (1-12)
-- **sale_quarter**: Quarter of sale (Q1-Q4)
-- **sale_year**: Year of sale (2022-2023)
-
-### Location Data (3 columns)
-- **zipcode**: ZIP code (10 different Seattle-area ZIP codes)
-- **lat**: Latitude coordinate
-- **long**: Longitude coordinate
-
-## Key Features for Analysis
-
-### 1. Price Drivers
-The dataset includes realistic relationships between price and features:
-- **Waterfront premium**: +30% average
-- **View quality**: +8% per rating point
-- **Property type**: Single Family and Multi-Family command premiums
-- **Neighborhood**: Waterfront (+25%) and Downtown (+15%) areas are most expensive
-- **School ratings**: Better schools correlate with higher prices
-- **Recent renovations**: +12% for properties renovated after 2015
-
-### 2. Market Segmentation
-Properties are naturally segmented into:
-- **Budget**: < $750K (25%)
-- **Mid-Range**: $750K - $1.2M (30%)
-- **Premium**: $1.2M - $1.8M (25%)
-- **Luxury**: > $1.8M (20%)
-
-### 3. Temporal Trends
-- Sales data spans 2 years (2022-2024)
-- Enables quarter-over-quarter and year-over-year analysis
-- Market velocity metrics through "days_on_market"
-
-### 4. Geographic Analysis
-- 10 different ZIP codes representing diverse neighborhoods
-- Lat/long coordinates for mapping and spatial analysis
-- Neighborhood-level characteristics (crime, walkability, schools)
-
-## Suggested Analysis & Visualizations
-
-### 1. Price Distribution & Trends
-- Price distribution by property type and neighborhood
-- Price trends over time (quarterly/yearly)
-- Price per square foot analysis
-
-### 2. Feature Impact Analysis
-- Impact of bedrooms, bathrooms on price
-- Premium analysis for waterfront, pool, renovations
-- School rating vs. price correlation
-
-### 3. Market Velocity
-- Days on market by price segment
-- Seasonal patterns in sales activity
-- HOA fees impact on marketability
-
-### 4. Geographic Insights
-- Price heatmaps by location
-- Neighborhood quality metrics visualization
-- ZIP code-level market analysis
-
-### 5. Property Characteristics
-- Condition vs. grade distribution
-- Age distribution and renovation patterns
-- Amenity combinations most common in different price ranges
-
-### 6. Comparative Analysis
-- Property type performance
-- Neighborhood comparisons (crime, schools, walkability)
-- Market segment characteristics
-
-## Default Dashboard Query
-The sample data comes with a pre-configured query that generates a comprehensive dashboard showcasing:
-1. KPI cards for key metrics (avg price, total listings)
-2. Price distribution by property type and neighborhood
-3. Quarterly price trends showing market evolution
-4. Feature impact analysis (bedrooms, waterfront, schools)
-5. Geographic price visualization
-6. Condition vs. grade matrix
-7. Market velocity by price segment
-
-## Data Quality
-- **No missing values**: All records are complete
-- **Realistic relationships**: Price adjustments based on actual market factors
-- **Diverse distribution**: Balanced representation across categories
-- **Time-series ready**: Proper date handling for temporal analysis
-- **GIS-ready**: Includes coordinates for mapping
-
-## Use Cases
-This dataset is ideal for demonstrating:
-- Real estate market analysis
-- Predictive pricing models
-- Market segmentation strategies
-- Investment opportunity identification
-- Neighborhood analysis and comparison
-- Temporal market trend analysis
-- Geographic price pattern detection
-

app/main.py

@@ -28,6 +28,7 @@ from .routes.forms import router as forms_router
 from .routes.responses import router as responses_router
 from .routes.analytics import router as analytics_router
 from .routes.users import router as users_router
+from .routes.uploads import router as uploads_router
 
 app = FastAPI(title="Backend", version="0.1.0")
 
@@ -107,6 +108,7 @@ app.include_router(forms_router)
 app.include_router(responses_router)
 app.include_router(analytics_router)
 app.include_router(users_router)
+app.include_router(uploads_router)
 
 # Initialize DB
 if os.getenv("AUTO_MIGRATE", "1") == "1":
@@ -123,4 +125,3 @@ if os.getenv("AUTO_MIGRATE", "1") == "1":
     except Exception as e:
         logging.error(f"Failed to initialize subscription plans: {e}")
 
-

app/models.py

@@ -83,14 +83,11 @@ class User(Base):
     created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow)
 
     subscriptions: Mapped[list["Subscription"]] = relationship(back_populates="user")
-    datasets: Mapped[list["Dataset"]] = relationship(back_populates="user")
     credits: Mapped["UserCredits"] = relationship(back_populates="user", uselist=False)
     credit_transactions: Mapped[list["CreditTransaction"]] = relationship(back_populates="user")
-    dashboard_queries: Mapped[list["DashboardQuery"]] = relationship(back_populates="user", cascade="all, delete-orphan")
-    chat_messages: Mapped[list["ChatMessage"]] = relationship(back_populates="user", cascade="all, delete-orphan")
-    public_dashboards: Mapped[list["PublicDashboard"]] = relationship(back_populates="user", cascade="all, delete-orphan")
     forms: Mapped[list["Form"]] = relationship(back_populates="user", cascade="all, delete-orphan")
     public_forms: Mapped[list["PublicForm"]] = relationship(back_populates="user", cascade="all, delete-orphan")
+    form_uploads: Mapped[list["FormUpload"]] = relationship(back_populates="user", cascade="all, delete-orphan")
 
 
 class Subscription(Base):
@@ -112,106 +109,6 @@ class Subscription(Base):
     plan: Mapped["SubscriptionPlan"] = relationship(back_populates="subscriptions")
 
 
-class Dataset(Base):
-    __tablename__ = "datasets"
-    
-    id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
-    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), index=True)
-    dataset_id: Mapped[str] = mapped_column(String(100), unique=True, index=True)
-    
-    # Basic metadata
-    filename: Mapped[str] = mapped_column(String(255))
-    row_count: Mapped[int] = mapped_column(Integer)
-    column_count: Mapped[int] = mapped_column(Integer)
-    file_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
-    
-    # Dataset context from DSPy (rich description)
-    context: Mapped[str | None] = mapped_column(Text, nullable=True)
-    context_generated_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
-    
-    # Column metadata as JSON
-    columns_info: Mapped[dict | None] = mapped_column(JSON, nullable=True)
-    # Store: {"columns": [...], "dtypes": {...}, "statistics": {...}}
-    
-    # Timestamps
-    created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow)
-    updated_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
-    
-    # Status tracking
-    context_status: Mapped[str] = mapped_column(String(50), default="pending")
-    # Values: "pending", "generating", "completed", "failed"
-    
-    user: Mapped[User] = relationship(back_populates="datasets")
-    dashboard_queries: Mapped[list["DashboardQuery"]] = relationship(back_populates="dataset", cascade="all, delete-orphan")
-    chat_messages: Mapped[list["ChatMessage"]] = relationship(back_populates="dataset", cascade="all, delete-orphan")
-    public_dashboards: Mapped[list["PublicDashboard"]] = relationship(back_populates="dataset", cascade="all, delete-orphan")
-
-
-class DashboardQuery(Base):
-    __tablename__ = "dashboard_queries"
-    
-    id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
-    dataset_id: Mapped[int] = mapped_column(ForeignKey("datasets.id"), index=True)
-    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), index=True)
-    
-    query: Mapped[str] = mapped_column(Text)  # User's request
-    query_type: Mapped[str] = mapped_column(String(50))  # "analyze", "edit", "add"
-    dashboard_title: Mapped[str | None] = mapped_column(String(255), nullable=True)
-    charts_data: Mapped[list[dict] | None] = mapped_column(JSON, nullable=True)  # Array of chart objects with code, figure, etc.
-    background_color: Mapped[str | None] = mapped_column(String(7), nullable=True, default="#ffffff")  # Hex color code
-    text_color: Mapped[str | None] = mapped_column(String(7), nullable=True, default="#1a1a1a")  # Hex color code for text
-    created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow, index=True)
-    
-    dataset: Mapped["Dataset"] = relationship(back_populates="dashboard_queries")
-    user: Mapped["User"] = relationship(back_populates="dashboard_queries")
-
-
-class ChatMessage(Base):
-    __tablename__ = "chat_messages"
-    
-    id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
-    dataset_id: Mapped[int] = mapped_column(ForeignKey("datasets.id"), index=True)
-    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), index=True)
-    
-    role: Mapped[str] = mapped_column(String(20))  # "user", "assistant"
-    content: Mapped[str] = mapped_column(Text)
-    query_type: Mapped[str | None] = mapped_column(String(50), nullable=True)  # "edit", "add", "data_analysis", "general_qa", "need_clarity", etc.
-    code: Mapped[str | None] = mapped_column(Text, nullable=True)  # Executable code if applicable
-    chart_index: Mapped[int | None] = mapped_column(Integer, nullable=True)  # If related to a chart
-    created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow, index=True)
-    
-    dataset: Mapped["Dataset"] = relationship(back_populates="chat_messages")
-    user: Mapped["User"] = relationship(back_populates="chat_messages")
-
-
-class PublicDashboard(Base):
-    __tablename__ = "public_dashboards"
-    
-    id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
-    dataset_id: Mapped[int] = mapped_column(ForeignKey("datasets.id"), index=True)
-    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), index=True)
-    
-    share_token: Mapped[str] = mapped_column(String(64), unique=True, index=True)
-    figures_data: Mapped[list[dict] | None] = mapped_column(JSON, nullable=True)  # Array of chart figures
-    dashboard_title: Mapped[str | None] = mapped_column(String(255), nullable=True)
-    background_color: Mapped[str | None] = mapped_column(String(7), nullable=True, default="#ffffff")  # Hex color code
-    text_color: Mapped[str | None] = mapped_column(String(7), nullable=True, default="#1a1a1a")  # Hex color code for text
-    background_opacity: Mapped[float | None] = mapped_column(Float, nullable=True, default=1.0)  # Background opacity
-    use_gradient: Mapped[bool | None] = mapped_column(Boolean, nullable=True, default=False)  # Use gradient background
-    gradient_color_2: Mapped[str | None] = mapped_column(String(7), nullable=True, default="#e5e7eb")  # Second gradient color
-    container_colors: Mapped[dict | None] = mapped_column(JSON, nullable=True)  # Container-specific colors
-    chart_colors: Mapped[dict | None] = mapped_column(JSON, nullable=True)  # Chart trace colors
-    chart_opacities: Mapped[dict | None] = mapped_column(JSON, nullable=True)  # Chart trace opacities
-    apply_to_containers: Mapped[bool | None] = mapped_column(Boolean, nullable=True, default=True)  # Apply colors to all containers
-    is_public: Mapped[bool] = mapped_column(Boolean, default=True)
-    expires_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
-    created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow)
-    updated_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
-    
-    dataset: Mapped["Dataset"] = relationship(back_populates="public_dashboards")
-    user: Mapped["User"] = relationship(back_populates="public_dashboards")
-
-
 class SubscriptionPlan(Base):
     __tablename__ = "subscription_plans"
     
@@ -297,6 +194,7 @@ class Form(Base):
     chat_messages_form: Mapped[list["ChatMessageForm"]] = relationship(back_populates="form", cascade="all, delete-orphan")
     versions: Mapped[list["FormVersion"]] = relationship(back_populates="form", cascade="all, delete-orphan")
     analytics_events: Mapped[list["FormAnalyticsEvent"]] = relationship(back_populates="form", cascade="all, delete-orphan")
+    uploads: Mapped[list["FormUpload"]] = relationship(back_populates="form", cascade="all, delete-orphan")
 
 
 class FormQuestion(Base):
@@ -322,6 +220,7 @@ class FormQuestion(Base):
     target_rules: Mapped[list["ConditionalRule"]] = relationship(foreign_keys="ConditionalRule.target_question_id", back_populates="target_question", cascade="all, delete-orphan")
     answers: Mapped[list["ResponseAnswer"]] = relationship(back_populates="question", cascade="all, delete-orphan")
     analytics_events: Mapped[list["FormAnalyticsEvent"]] = relationship(back_populates="question", cascade="all, delete-orphan")
+    uploads: Mapped[list["FormUpload"]] = relationship(back_populates="question", cascade="all, delete-orphan")
 
 
 class ConditionalRule(Base):
@@ -381,6 +280,7 @@ class FormResponse(Base):
     form: Mapped[Form] = relationship(back_populates="responses")
     answers: Mapped[list["ResponseAnswer"]] = relationship(back_populates="response", cascade="all, delete-orphan")
     analytics_events: Mapped[list["FormAnalyticsEvent"]] = relationship(back_populates="submission", cascade="all, delete-orphan")
+    uploads: Mapped[list["FormUpload"]] = relationship(back_populates="response", cascade="all, delete-orphan")
 
 
 class ResponseAnswer(Base):
@@ -400,6 +300,32 @@ class ResponseAnswer(Base):
     question: Mapped[FormQuestion] = relationship(back_populates="answers")
 
 
+class FormUpload(Base):
+    """Uploaded files for form responses"""
+    __tablename__ = "form_uploads"
+    
+    id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
+    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), index=True)
+    form_id: Mapped[int] = mapped_column(ForeignKey("forms.id"), index=True)
+    form_response_id: Mapped[int | None] = mapped_column(ForeignKey("form_responses.id"), nullable=True, index=True)
+    form_question_id: Mapped[int] = mapped_column(ForeignKey("form_questions.id"), index=True)
+    
+    status: Mapped[str] = mapped_column(String(20), default="pending", index=True)
+    original_filename: Mapped[str] = mapped_column(String(512))
+    content_type: Mapped[str | None] = mapped_column(String(255), nullable=True)
+    size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    s3_key: Mapped[str] = mapped_column(String(1024), unique=True, index=True)
+    
+    created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow)
+    uploaded_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
+    attached_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
+    
+    user: Mapped[User] = relationship(back_populates="form_uploads")
+    form: Mapped[Form] = relationship(back_populates="uploads")
+    response: Mapped["FormResponse"] = relationship(back_populates="uploads")
+    question: Mapped[FormQuestion] = relationship(back_populates="uploads")
+
+
 class PublicForm(Base):
     """Public shareable forms"""
     __tablename__ = "public_forms"
@@ -538,4 +464,3 @@ class WebhookConfig(Base):
     # Relationships
     form: Mapped[Form] = relationship()
     user: Mapped[User] = relationship()
-

app/routes/auth.py

@@ -45,7 +45,7 @@ def get_current_user_info(
 ):
     """Get current authenticated user's profile information"""
     # Get user's subscription info
-    from ..models import Subscription, Dataset
+    from ..models import Subscription, Form
     from datetime import datetime, timezone
     
     subscription = db.query(Subscription).filter(
@@ -62,13 +62,13 @@ def get_current_user_info(
             "created_at": subscription.created_at.isoformat()
         }
     
-    # Calculate dashboards created this month
+    # Calculate forms created this month
     now = datetime.now(timezone.utc)
     start_of_month = now.replace(day=1, hour=0, minute=0, second=0, microsecond=0)
     
-    dashboards_this_month = db.query(Dataset).filter(
-        Dataset.user_id == current_user.id,
-        Dataset.created_at >= start_of_month
+    forms_this_month = db.query(Form).filter(
+        Form.user_id == current_user.id,
+        Form.created_at >= start_of_month
     ).count()
     
     return {
@@ -79,7 +79,7 @@ def get_current_user_info(
         "provider": current_user.provider,
         "is_active": current_user.is_active,
         "created_at": current_user.created_at.isoformat(),
-        "dashboards_this_month": dashboards_this_month,
+        "forms_this_month": forms_this_month,
         "subscription": subscription_info or {
             "tier": "free",
             "status": "inactive",

app/routes/plans.py

@@ -15,8 +15,6 @@ router = APIRouter(prefix="/api/plans", tags=["plans"])
 
 class PlanFeatures(BaseModel):
     """Plan features model"""
-    max_datasets: int | None = None
-    max_file_size_mb: int | None = None
     export_formats: List[str] | None = None
     priority_support: bool | None = None
     custom_branding: bool | None = None

app/routes/responses.py

@@ -16,7 +16,7 @@ from datetime import datetime
 
 from ..core.db import get_db
 from ..core.security import get_current_user
-from ..models import User, Form, FormQuestion, FormResponse, ResponseAnswer, PublicForm
+from ..models import User, Form, FormQuestion, FormResponse, ResponseAnswer, PublicForm, FormUpload, QuestionType
 from ..schemas.form import (
     SubmissionCreate, SubmissionResponse,
     FormResponseDetail, FormResponsesList, ResponseAnswerDetail,
@@ -26,6 +26,7 @@ from ..schemas.validation import PartialSubmissionCreate, AutoSaveResponse
 from ..services.validation_service import validation_service
 from ..services.analytics_service import analytics_service
 from ..services.webhook_service import webhook_service
+from ..services.s3_service import s3_service
 from ..middleware.rate_limiter import rate_limiter
 
 import logging
@@ -35,6 +36,116 @@ logger = logging.getLogger(__name__)
 router = APIRouter(prefix="/api", tags=["responses"])
 
 
+def _extract_upload_ids(answer_value: dict) -> List[int]:
+    if not isinstance(answer_value, dict):
+        return []
+    files = answer_value.get("files") or []
+    upload_ids = []
+    for item in files:
+        upload_id = None
+        if isinstance(item, dict):
+            upload_id = item.get("upload_id")
+        elif isinstance(item, int):
+            upload_id = item
+        elif isinstance(item, str) and item.isdigit():
+            upload_id = int(item)
+        if isinstance(upload_id, int):
+            upload_ids.append(upload_id)
+    return upload_ids
+
+
+def _attach_uploads(
+    db: Session,
+    upload_ids: List[int],
+    form_id: int,
+    response_id: int,
+    question_id: int
+) -> None:
+    if not upload_ids:
+        return
+    if not question_id:
+        return
+    try:
+        question_id = int(question_id)
+    except (TypeError, ValueError):
+        return
+    uploads = db.query(FormUpload).filter(FormUpload.id.in_(upload_ids)).all()
+    upload_map = {upload.id: upload for upload in uploads}
+    
+    for upload_id in upload_ids:
+        upload = upload_map.get(upload_id)
+        if not upload:
+            raise HTTPException(
+                status_code=status.HTTP_400_BAD_REQUEST,
+                detail="Invalid upload reference"
+            )
+        if upload.form_id != form_id or upload.form_question_id != question_id:
+            raise HTTPException(
+                status_code=status.HTTP_400_BAD_REQUEST,
+                detail="Upload does not match form or question"
+            )
+        if upload.status not in ["pending", "uploaded", "attached"]:
+            raise HTTPException(
+                status_code=status.HTTP_400_BAD_REQUEST,
+                detail="Upload is not ready"
+            )
+        
+        upload.form_response_id = response_id
+        upload.status = "attached"
+        upload.attached_at = datetime.utcnow()
+
+
+def _enrich_answer_files(
+    answer_value: dict,
+    upload_map: dict,
+    include_download_url: bool = True
+) -> dict:
+    if not isinstance(answer_value, dict):
+        return answer_value
+    files = answer_value.get("files") or []
+    if not files:
+        return answer_value
+    
+    enriched = []
+    for item in files:
+        upload_id = None
+        if isinstance(item, dict):
+            upload_id = item.get("upload_id")
+        elif isinstance(item, int):
+            upload_id = item
+        elif isinstance(item, str) and item.isdigit():
+            upload_id = int(item)
+        
+        if not isinstance(upload_id, int):
+            continue
+        
+        upload = upload_map.get(upload_id)
+        if not upload:
+            continue
+        
+        file_entry = {
+            "upload_id": upload.id,
+            "filename": upload.original_filename,
+            "content_type": upload.content_type,
+            "size_bytes": upload.size_bytes,
+            "s3_key": upload.s3_key
+        }
+        
+        if include_download_url:
+            try:
+                file_entry["download_url"] = s3_service.generate_presigned_download_url(
+                    key=upload.s3_key,
+                    filename=upload.original_filename
+                )
+            except Exception:
+                pass
+        
+        enriched.append(file_entry)
+    
+    updated = dict(answer_value)
+    updated["files"] = enriched
+    return updated
+
 @router.post("/public/forms/{token}/submit", response_model=SubmissionResponse)
 async def submit_form(
     token: str,
@@ -78,23 +189,83 @@ async def submit_form(
     client_ip = request.client.host if request.client else None
     
     try:
-        # Create form response
-        form_response = FormResponse(
-            form_id=form.id,
-            ip_address=client_ip,
-            submission_metadata=submission.metadata or {}
-        )
-        db.add(form_response)
-        db.flush()
+        # Try to reuse existing in-progress response for this session (if provided)
+        form_response = None
+        if submission.session_id:
+            form_response = db.query(FormResponse).filter(
+                FormResponse.form_id == form.id,
+                FormResponse.session_id == submission.session_id,
+                FormResponse.status.in_(["in_progress", "partial"])
+            ).first()
+        
+        if form_response:
+            form_response.last_updated_at = datetime.utcnow()
+            form_response.submitted_at = datetime.utcnow()
+            form_response.ip_address = client_ip
+            form_response.utm_source = submission.utm_source
+            form_response.utm_medium = submission.utm_medium
+            form_response.utm_campaign = submission.utm_campaign
+            if submission.metadata:
+                form_response.submission_metadata = submission.metadata
+        else:
+            # Create new form response
+            form_response = FormResponse(
+                form_id=form.id,
+                session_id=submission.session_id,
+                status="complete",
+                form_version=submission.form_version,
+                started_at=datetime.utcnow(),
+                submitted_at=datetime.utcnow(),
+                ip_address=client_ip,
+                utm_source=submission.utm_source,
+                utm_medium=submission.utm_medium,
+                utm_campaign=submission.utm_campaign,
+                submission_metadata=submission.metadata or {}
+            )
+            db.add(form_response)
+            db.flush()
         
-        # Create answers
+        # Create/update answers
         for answer_data in submission.answers:
-            answer = ResponseAnswer(
-                form_response_id=form_response.id,
-                form_question_id=answer_data.question_id,
-                answer_value=answer_data.answer_value.model_dump()
+            answer_value = answer_data.answer_value.model_dump()
+            upload_ids = _extract_upload_ids(answer_value)
+            _attach_uploads(
+                db=db,
+                upload_ids=upload_ids,
+                form_id=form.id,
+                response_id=form_response.id,
+                question_id=answer_data.question_id
             )
-            db.add(answer)
+            
+            existing_answer = db.query(ResponseAnswer).filter(
+                ResponseAnswer.form_response_id == form_response.id,
+                ResponseAnswer.form_question_id == answer_data.question_id
+            ).first()
+            
+            if existing_answer:
+                existing_answer.answer_value = answer_value
+            else:
+                answer = ResponseAnswer(
+                    form_response_id=form_response.id,
+                    form_question_id=answer_data.question_id,
+                    answer_value=answer_value
+                )
+                db.add(answer)
+        
+        # Validate and set status based on whether all required fields are answered
+        validation_result = validation_service.validate_submission(
+            db=db,
+            form_id=form.id,
+            answers=[
+                {"question_id": a.question_id, "answer_value": a.answer_value.model_dump()}
+                for a in submission.answers
+            ],
+            is_complete=True,
+            form_version=submission.form_version
+        )
+        
+        # Set status: "complete" if all required fields answered, "partial" otherwise
+        form_response.status = "complete" if validation_result.is_valid else "partial"
         
         db.commit()
         db.refresh(form_response)
@@ -106,6 +277,9 @@ async def submit_form(
             message=public_form.custom_thank_you_message or "Thank you for your submission!"
         )
         
+    except HTTPException:
+        db.rollback()
+        raise
     except Exception as e:
         db.rollback()
         logger.error(f"Form submission failed: {e}")
@@ -171,15 +345,25 @@ async def autosave_submission(
                     ResponseAnswer.form_question_id == answer_data.get('question_id')
                 ).first()
                 
+                answer_value = answer_data.get('answer_value', {})
+                upload_ids = _extract_upload_ids(answer_value)
+                _attach_uploads(
+                    db=db,
+                    upload_ids=upload_ids,
+                    form_id=form.id,
+                    response_id=existing_response.id,
+                    question_id=answer_data.get('question_id')
+                )
+                
                 if existing_answer:
                     # Update existing answer
-                    existing_answer.answer_value = answer_data.get('answer_value', {})
+                    existing_answer.answer_value = answer_value
                 else:
                     # Create new answer
                     new_answer = ResponseAnswer(
                         form_response_id=existing_response.id,
                         form_question_id=answer_data.get('question_id'),
-                        answer_value=answer_data.get('answer_value', {})
+                        answer_value=answer_value
                     )
                     db.add(new_answer)
             
@@ -205,10 +389,19 @@ async def autosave_submission(
             
             # Create answers
             for answer_data in submission.answers:
+                answer_value = answer_data.get('answer_value', {})
+                upload_ids = _extract_upload_ids(answer_value)
+                _attach_uploads(
+                    db=db,
+                    upload_ids=upload_ids,
+                    form_id=form.id,
+                    response_id=form_response.id,
+                    question_id=answer_data.get('question_id')
+                )
                 answer = ResponseAnswer(
                     form_response_id=form_response.id,
                     form_question_id=answer_data.get('question_id'),
-                    answer_value=answer_data.get('answer_value', {})
+                    answer_value=answer_value
                 )
                 db.add(answer)
         
@@ -230,6 +423,9 @@ async def autosave_submission(
             message="Auto-saved successfully"
         )
         
+    except HTTPException:
+        db.rollback()
+        raise
     except Exception as e:
         db.rollback()
         logger.error(f"Auto-save failed: {e}")
@@ -314,6 +510,16 @@ async def get_form_responses(
         FormResponse.form_id == form_id
     ).order_by(desc(FormResponse.submitted_at)).limit(limit).offset(offset).all()
     
+    upload_ids = set()
+    for response in responses:
+        for answer in response.answers:
+            upload_ids.update(_extract_upload_ids(answer.answer_value or {}))
+    
+    upload_map = {}
+    if upload_ids:
+        uploads = db.query(FormUpload).filter(FormUpload.id.in_(upload_ids)).all()
+        upload_map = {upload.id: upload for upload in uploads}
+    
     # Build detailed response list
     response_details = []
     for response in responses:
@@ -324,16 +530,22 @@ async def get_form_responses(
             ).first()
             
             if question:
+                answer_value = _enrich_answer_files(
+                    answer.answer_value or {},
+                    upload_map,
+                    include_download_url=True
+                )
                 answers.append(ResponseAnswerDetail(
                     question_id=question.id,
                     question_text=question.question_text,
                     question_type=question.question_type,
-                    answer_value=answer.answer_value or {}
+                    answer_value=answer_value
                 ))
         
         response_details.append(FormResponseDetail(
             id=response.id,
             form_id=response.form_id,
+            status=response.status,
             submitted_at=response.submitted_at,
             ip_address=response.ip_address,
             answers=answers
@@ -378,17 +590,31 @@ async def get_single_response(
     
     # Build answer details
     answers = []
+    upload_ids = set()
+    for answer in response.answers:
+        upload_ids.update(_extract_upload_ids(answer.answer_value or {}))
+    
+    upload_map = {}
+    if upload_ids:
+        uploads = db.query(FormUpload).filter(FormUpload.id.in_(upload_ids)).all()
+        upload_map = {upload.id: upload for upload in uploads}
+    
     for answer in response.answers:
         question = db.query(FormQuestion).filter(
             FormQuestion.id == answer.form_question_id
         ).first()
         
         if question:
+            answer_value = _enrich_answer_files(
+                answer.answer_value or {},
+                upload_map,
+                include_download_url=True
+            )
             answers.append(ResponseAnswerDetail(
                 question_id=question.id,
                 question_text=question.question_text,
                 question_type=question.question_type,
-                answer_value=answer.answer_value or {}
+                answer_value=answer_value
             ))
     
     return FormResponseDetail(
@@ -403,6 +629,7 @@ async def get_single_response(
 @router.get("/forms/{form_id}/responses/export/csv")
 async def export_responses_csv(
     form_id: int,
+    request: Request,
     current_user: User = Depends(get_current_user),
     db: Session = Depends(get_db)
 ):
@@ -431,15 +658,40 @@ async def export_responses_csv(
         FormResponse.form_id == form_id
     ).order_by(desc(FormResponse.submitted_at)).all()
     
+    upload_ids = set()
+    for response in responses:
+        for answer in response.answers:
+            upload_ids.update(_extract_upload_ids(answer.answer_value or {}))
+    
+    upload_map = {}
+    if upload_ids:
+        uploads = db.query(FormUpload).filter(FormUpload.id.in_(upload_ids)).all()
+        upload_map = {upload.id: upload for upload in uploads}
+    
+    upload_ids = set()
+    for response in responses:
+        for answer in response.answers:
+            upload_ids.update(_extract_upload_ids(answer.answer_value or {}))
+    
+    upload_map = {}
+    if upload_ids:
+        uploads = db.query(FormUpload).filter(FormUpload.id.in_(upload_ids)).all()
+        upload_map = {upload.id: upload for upload in uploads}
+    
     # Create CSV in memory
     output = io.StringIO()
     writer = csv.writer(output)
     
     # Write headers
     headers = ["Response ID", "Submitted At", "IP Address"]
-    headers.extend([q.question_text for q in questions])
+    for question in questions:
+        headers.append(question.question_text)
+        if question.question_type == QuestionType.FILE_UPLOAD:
+            headers.append(f"{question.question_text} (download_url)")
     writer.writerow(headers)
     
+    base_url = str(request.base_url).rstrip("/")
+
     # Write data rows
     for response in responses:
         row = [
@@ -457,6 +709,12 @@ async def export_responses_csv(
         # Add answers in question order
         for question in questions:
             answer_value = answer_map.get(question.id, {})
+            if isinstance(answer_value, dict):
+                answer_value = _enrich_answer_files(
+                    answer_value,
+                    upload_map,
+                    include_download_url=True
+                )
             
             # Format answer based on type
             if isinstance(answer_value, dict):
@@ -471,10 +729,42 @@ async def export_responses_csv(
                     row.append(answer_value["date"])
                 elif "rating" in answer_value:
                     row.append(str(answer_value["rating"]))
+                elif "files" in answer_value and answer_value["files"]:
+                    file_labels = []
+                    file_links = []
+                    for file_item in answer_value["files"]:
+                        if not isinstance(file_item, dict):
+                            file_labels.append(str(file_item))
+                            file_links.append(str(file_item))
+                            continue
+                        file_labels.append(
+                            file_item.get("download_url")
+                            or file_item.get("filename")
+                            or file_item.get("original_filename")
+                            or file_item.get("s3_key")
+                            or str(file_item.get("upload_id", ""))
+                        )
+                        if file_item.get("download_url"):
+                            file_links.append(file_item["download_url"])
+                        elif file_item.get("upload_id"):
+                            file_links.append(
+                                f"{base_url}/api/forms/{form_id}/uploads/{file_item['upload_id']}/download"
+                            )
+                    row.append(", ".join([label for label in file_labels if label]))
+                    if question.question_type == QuestionType.FILE_UPLOAD:
+                        row.append(", ".join([link for link in file_links if link]))
+                elif "file_url" in answer_value:
+                    row.append(answer_value["file_url"])
+                    if question.question_type == QuestionType.FILE_UPLOAD:
+                        row.append(answer_value["file_url"])
                 else:
                     row.append(json.dumps(answer_value))
+                    if question.question_type == QuestionType.FILE_UPLOAD:
+                        row.append("")
             else:
                 row.append(str(answer_value))
+                if question.question_type == QuestionType.FILE_UPLOAD:
+                    row.append("")
         
         writer.writerow(row)
     
@@ -493,6 +783,7 @@ async def export_responses_csv(
 @router.get("/forms/{form_id}/responses/export/json")
 async def export_responses_json(
     form_id: int,
+    request: Request,
     current_user: User = Depends(get_current_user),
     db: Session = Depends(get_db)
 ):
@@ -516,6 +807,18 @@ async def export_responses_json(
         FormResponse.form_id == form_id
     ).order_by(desc(FormResponse.submitted_at)).all()
     
+    upload_ids = set()
+    for response in responses:
+        for answer in response.answers:
+            upload_ids.update(_extract_upload_ids(answer.answer_value or {}))
+    
+    upload_map = {}
+    if upload_ids:
+        uploads = db.query(FormUpload).filter(FormUpload.id.in_(upload_ids)).all()
+        upload_map = {upload.id: upload for upload in uploads}
+    
+    base_url = str(request.base_url).rstrip("/")
+
     # Build export data
     export_data = {
         "form_id": form_id,
@@ -539,11 +842,30 @@ async def export_responses_json(
             ).first()
             
             if question:
+                answer_value = _enrich_answer_files(
+                    answer.answer_value or {},
+                    upload_map,
+                    include_download_url=True
+                )
+                if isinstance(answer_value, dict) and "files" in answer_value:
+                    updated_files = []
+                    for file_item in answer_value.get("files") or []:
+                        if not isinstance(file_item, dict):
+                            updated_files.append(file_item)
+                            continue
+                        if not file_item.get("download_url") and file_item.get("upload_id"):
+                            file_item = dict(file_item)
+                            file_item["download_url"] = (
+                                f"{base_url}/api/forms/{form_id}/uploads/{file_item['upload_id']}/download"
+                            )
+                        updated_files.append(file_item)
+                    answer_value = dict(answer_value)
+                    answer_value["files"] = updated_files
                 response_data["answers"].append({
                     "question_id": question.id,
                     "question_text": question.question_text,
                     "question_type": question.question_type.value,
-                    "answer_value": answer.answer_value
+                    "answer_value": answer_value
                 })
         
         export_data["responses"].append(response_data)
@@ -591,4 +913,3 @@ async def delete_response(
     db.commit()
     
     return None
-

app/routes/uploads.py

@@ -0,0 +1,213 @@
+"""
+Uploads API Routes
+==================
+
+Presigned S3 uploads for public form submissions.
+"""
+
+import os
+from datetime import datetime
+from fastapi import APIRouter, Depends, HTTPException, status
+from sqlalchemy.orm import Session
+
+from ..core.db import get_db
+from ..core.security import get_current_user
+from ..models import Form, FormQuestion, FormUpload, PublicForm, QuestionType, User
+from ..schemas.uploads import (
+    UploadCreateRequest,
+    UploadCreateResponse,
+    UploadCompleteRequest,
+    UploadCompleteResponse
+)
+from ..services.s3_service import s3_service
+
+router = APIRouter(prefix="/api", tags=["uploads"])
+
+
+@router.post("/public/forms/{token}/uploads", response_model=UploadCreateResponse)
+async def create_public_upload(
+    token: str,
+    payload: UploadCreateRequest,
+    db: Session = Depends(get_db)
+):
+    """Create a presigned upload URL for a public form submission."""
+    public_form = db.query(PublicForm).filter(
+        PublicForm.share_token == token,
+        PublicForm.is_public == True
+    ).first()
+    
+    if not public_form:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail="Form not found or no longer accepting responses"
+        )
+    
+    if public_form.expires_at and public_form.expires_at < datetime.utcnow():
+        raise HTTPException(
+            status_code=status.HTTP_410_GONE,
+            detail="This form has expired"
+        )
+    
+    form = db.query(Form).filter(Form.id == public_form.form_id).first()
+    if not form:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail="Form not found"
+        )
+    
+    question = db.query(FormQuestion).filter(
+        FormQuestion.id == payload.question_id,
+        FormQuestion.form_id == form.id
+    ).first()
+    
+    if not question:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail="Question not found"
+        )
+    
+    if question.question_type != QuestionType.FILE_UPLOAD:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail="Question does not accept file uploads"
+        )
+    
+    settings = question.settings or {}
+    max_size = settings.get("max_file_size")
+    if max_size and payload.size_bytes and payload.size_bytes > max_size:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail="File exceeds maximum allowed size"
+        )
+    
+    allowed_types = settings.get("file_types") or []
+    if allowed_types:
+        ext = os.path.splitext(payload.filename)[1].lower()
+        allowed = {t.lower() for t in allowed_types}
+        if ext and ext not in allowed:
+            raise HTTPException(
+                status_code=status.HTTP_400_BAD_REQUEST,
+                detail="File type not allowed"
+            )
+    
+    upload = FormUpload(
+        user_id=public_form.user_id,
+        form_id=form.id,
+        form_question_id=question.id,
+        original_filename=payload.filename,
+        content_type=payload.content_type,
+        size_bytes=payload.size_bytes,
+        status="pending",
+        s3_key=""
+    )
+    db.add(upload)
+    db.flush()
+    
+    s3_key = s3_service.build_key(form.id, form.title, upload.id, payload.filename)
+    upload.s3_key = s3_key
+    db.commit()
+    
+    try:
+        upload_url = s3_service.generate_presigned_upload_url(
+            key=s3_key,
+            content_type=payload.content_type
+        )
+    except Exception:
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="S3 upload is not configured"
+        )
+    
+    return UploadCreateResponse(
+        upload_id=upload.id,
+        upload_url=upload_url,
+        s3_key=s3_key,
+        expires_in=900,
+        original_filename=payload.filename
+    )
+
+
+@router.post("/public/forms/{token}/uploads/{upload_id}/complete", response_model=UploadCompleteResponse)
+async def complete_public_upload(
+    token: str,
+    upload_id: int,
+    payload: UploadCompleteRequest,
+    db: Session = Depends(get_db)
+):
+    """Mark a public upload as completed."""
+    public_form = db.query(PublicForm).filter(
+        PublicForm.share_token == token,
+        PublicForm.is_public == True
+    ).first()
+    
+    if not public_form:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail="Form not found"
+        )
+    
+    upload = db.query(FormUpload).filter(
+        FormUpload.id == upload_id,
+        FormUpload.form_id == public_form.form_id
+    ).first()
+    
+    if not upload:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail="Upload not found"
+        )
+    
+    if payload.content_type:
+        upload.content_type = payload.content_type
+    if payload.size_bytes is not None:
+        upload.size_bytes = payload.size_bytes
+    
+    upload.status = "uploaded"
+    upload.uploaded_at = datetime.utcnow()
+    db.commit()
+    
+    return UploadCompleteResponse(success=True)
+
+
+@router.get("/forms/{form_id}/uploads/{upload_id}/download")
+async def download_upload(
+    form_id: int,
+    upload_id: int,
+    current_user: User = Depends(get_current_user),
+    db: Session = Depends(get_db)
+):
+    """Generate a presigned download URL for a file upload (owner only)."""
+    form = db.query(Form).filter(
+        Form.id == form_id,
+        Form.user_id == current_user.id
+    ).first()
+    
+    if not form:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail="Form not found"
+        )
+    
+    upload = db.query(FormUpload).filter(
+        FormUpload.id == upload_id,
+        FormUpload.form_id == form_id
+    ).first()
+    
+    if not upload:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail="Upload not found"
+        )
+    
+    try:
+        download_url = s3_service.generate_presigned_download_url(
+            key=upload.s3_key,
+            filename=upload.original_filename
+        )
+    except Exception:
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="S3 download is not configured"
+        )
+    
+    return {"download_url": download_url}

app/schemas/chat.py

@@ -1,34 +0,0 @@
-from pydantic import BaseModel
-from typing import Optional, Dict, Any, Literal
-
-
-class ChatRequest(BaseModel):
-    message: str
-    dataset_id: Optional[str] = None
-    fig_data: Optional[Dict[str, Any]] = None
-    plotly_code: Optional[str] = None
-
-
-class ChatResponse(BaseModel):
-    reply: str
-    user: str | None = None
-    matched_chart: Optional[Dict[str, Any]] = None
-    code_type: Optional[Literal['plotly_edit', 'analysis', 'add_chart_query']] = None
-    executable_code: Optional[str] = None
-    query_type: Optional[str] = None  # The detected query type (for showing action buttons on need_clarity)
-
-
-class FixVisualizationRequest(BaseModel):
-    plotly_code: str
-    error_message: str
-    dataset_id: Optional[str] = None
-
-
-class RetryCodeRequest(BaseModel):
-    user_query: str
-    dataset_id: str
-    code_type: str  # 'plotly_edit' or 'analysis'
-    plotly_code: Optional[str] = None  # Only for plotly_edit
-    data_context: str
-
-

app/schemas/form.py

@@ -153,6 +153,7 @@ class AnswerValue(BaseModel):
     date: Optional[str] = None  # For date/time
     choices: Optional[List[str]] = None  # For multiple choice, checkboxes
     file_url: Optional[str] = None  # For file uploads
+    files: Optional[List[Dict[str, Any]]] = None  # For file uploads
     rating: Optional[int] = None  # For ratings
     signature: Optional[str] = None  # Base64 signature image
     wallet_address: Optional[str] = None  # For wallet connect
@@ -172,6 +173,11 @@ class SubmissionCreate(BaseModel):
     """Schema for submitting a form response"""
     answers: List[SubmitAnswer]
     metadata: Optional[Dict[str, Any]] = None
+    session_id: Optional[str] = None
+    form_version: Optional[int] = None
+    utm_source: Optional[str] = None
+    utm_medium: Optional[str] = None
+    utm_campaign: Optional[str] = None
 
 
 class SubmissionResponse(BaseModel):
@@ -200,6 +206,7 @@ class FormResponseDetail(BaseModel):
     """Detailed view of a single form response"""
     id: int
     form_id: int
+    status: Optional[str] = None
     submitted_at: datetime
     ip_address: Optional[str]
     answers: List[ResponseAnswerDetail]

app/schemas/uploads.py

@@ -0,0 +1,26 @@
+from pydantic import BaseModel, Field
+from typing import Optional
+
+
+class UploadCreateRequest(BaseModel):
+    question_id: int = Field(..., ge=1)
+    filename: str = Field(..., min_length=1, max_length=512)
+    content_type: Optional[str] = None
+    size_bytes: Optional[int] = Field(default=None, ge=0)
+
+
+class UploadCreateResponse(BaseModel):
+    upload_id: int
+    upload_url: str
+    s3_key: str
+    expires_in: int
+    original_filename: str
+
+
+class UploadCompleteRequest(BaseModel):
+    content_type: Optional[str] = None
+    size_bytes: Optional[int] = Field(default=None, ge=0)
+
+
+class UploadCompleteResponse(BaseModel):
+    success: bool

app/services/agents.py

@@ -106,7 +106,7 @@ class FormPlannerSignature(dspy.Signature):
     6. Choose appropriate component types based on the data being collected
     7. Mark essential components as required
     8. Order components logically (general to specific)
-    9. Include 3-5 components typically
+    9. **INCLUDE AT LEAST 5 COMPONENTS** - Forms should have a minimum of 5 questions/components. You can reuse the same component type multiple times if appropriate (e.g., multiple short_answer fields for different information)
     10. Add conditional logic where it makes sense (e.g., show follow-up based on previous answer)
     11. Generate component_ids like "comp_1", "comp_2", etc.
     12. **PREFER STRUCTURED INPUTS** over text when possible (use checkboxes, dropdowns, ratings instead of open text)

app/services/export_service.py

@@ -445,6 +445,21 @@ class ExportService:
         if "ranked_items" in answer_value and answer_value["ranked_items"]:
             return ", ".join(answer_value["ranked_items"])
         
+        # File uploads
+        if "files" in answer_value and answer_value["files"]:
+            file_labels = []
+            for file_item in answer_value["files"]:
+                if not isinstance(file_item, dict):
+                    file_labels.append(str(file_item))
+                    continue
+                file_labels.append(
+                    file_item.get("filename")
+                    or file_item.get("original_filename")
+                    or file_item.get("s3_key")
+                    or str(file_item.get("upload_id", ""))
+                )
+            return ", ".join([label for label in file_labels if label])
+        
         # File upload
         if "file_url" in answer_value and answer_value["file_url"]:
             return answer_value["file_url"]

app/services/plan_service.py

@@ -215,9 +215,7 @@ class PlanService:
                 "stripe_price_id_yearly": os.getenv("STRIPE_FREE_PRICE_YEARLY_ID"),
                 "stripe_product_id": os.getenv("STRIPE_FREE_PRODUCT_ID"),
                 "features": {
-                    "max_datasets": 3,
-                    "max_file_size_mb": 10,
-                    "export_formats": ["png", "csv"]
+                    "export_formats": ["csv", "json"]
                 },
                 "sort_order": 0
             },
@@ -233,9 +231,7 @@ class PlanService:
                 "stripe_price_id_yearly": os.getenv("STRIPE_PRO_PRICE_YEARLY_ID"),
                 "stripe_product_id": os.getenv("STRIPE_PRO_PRODUCT_ID"),
                 "features": {
-                    "max_datasets": 50,
-                    "max_file_size_mb": 100,
-                    "export_formats": ["png", "csv", "pdf", "xlsx"],
+                    "export_formats": ["csv", "json", "pdf"],
                     "priority_support": True
                 },
                 "sort_order": 1
@@ -252,9 +248,7 @@ class PlanService:
                 "stripe_price_id_yearly": os.getenv("STRIPE_ULTRA_PRICE_YEARLY_ID"),
                 "stripe_product_id": os.getenv("STRIPE_ULTRA_PRODUCT_ID"),
                 "features": {
-                    "max_datasets": -1,  # Unlimited
-                    "max_file_size_mb": 500,
-                    "export_formats": ["png", "csv", "pdf", "xlsx", "json"],
+                    "export_formats": ["csv", "json", "pdf"],
                     "priority_support": True,
                     "custom_branding": True,
                     "api_access": True

app/services/s3_service.py

@@ -0,0 +1,98 @@
+import os
+import re
+from typing import Optional
+import boto3
+
+
+class S3Service:
+    """S3 utility for presigned uploads/downloads and key generation."""
+
+    def __init__(self):
+        self.bucket = os.getenv("S3_BUCKET_NAME")
+
+        region = os.getenv("AWS_REGION")
+        access_key = os.getenv("AWS_ACCESS_KEY_ID")
+        secret_key = os.getenv("AWS_SECRET_ACCESS_KEY")
+
+        self.client = boto3.client(
+            "s3",
+            region_name=region,
+            aws_access_key_id=access_key,
+            aws_secret_access_key=secret_key
+        )
+
+        app_env = os.getenv("APP_ENV", "dev")
+        self.prefix = os.getenv("S3_PREFIX", f"autoform/{app_env}").strip("/")
+
+    def build_key(self, form_id: int, form_title: str, upload_id: int, filename: str) -> str:
+        form_slug = self._slugify(form_title)
+        filename_safe = self._sanitize_filename(filename)
+
+        folder = f"f{form_id}"
+        if form_slug:
+            folder = f"{folder}-{form_slug}"
+
+        return f"{self.prefix}/{folder}/{upload_id}__{filename_safe}"
+
+    def generate_presigned_upload_url(
+        self,
+        key: str,
+        content_type: Optional[str],
+        expires_in: int = 900
+    ) -> str:
+        if not self.bucket:
+            raise ValueError("S3_BUCKET_NAME is not set")
+        params = {
+            "Bucket": self.bucket,
+            "Key": key
+        }
+
+        if content_type:
+            params["ContentType"] = content_type
+
+        return self.client.generate_presigned_url(
+            "put_object",
+            Params=params,
+            ExpiresIn=expires_in
+        )
+
+    def generate_presigned_download_url(
+        self,
+        key: str,
+        filename: Optional[str],
+        expires_in: int = 900
+    ) -> str:
+        if not self.bucket:
+            raise ValueError("S3_BUCKET_NAME is not set")
+        params = {
+            "Bucket": self.bucket,
+            "Key": key
+        }
+
+        if filename:
+            safe_name = self._sanitize_filename(filename)
+            params["ResponseContentDisposition"] = f'attachment; filename="{safe_name}"'
+
+        return self.client.generate_presigned_url(
+            "get_object",
+            Params=params,
+            ExpiresIn=expires_in
+        )
+
+    def _slugify(self, value: str) -> str:
+        if not value:
+            return ""
+
+        value = value.lower()
+        value = re.sub(r"[^a-z0-9]+", "-", value)
+        value = value.strip("-")
+        return value[:60]
+
+    def _sanitize_filename(self, filename: str) -> str:
+        base = os.path.basename(filename or "")
+        base = base.replace(" ", "_")
+        base = re.sub(r"[^A-Za-z0-9._-]", "", base)
+        return base[:120] or "file"
+
+
+s3_service = S3Service()

app/services/validation_service.py

@@ -77,13 +77,19 @@ class ValidationService:
         
         # Create answer map for quick lookup
         answer_map = {ans.get('question_id'): ans.get('answer_value', {}) for ans in answers}
+        visible_question_ids = self._get_visible_question_ids(form, answer_map)
         
         # Validate each question
         for question in form.questions:
             answer_value = answer_map.get(question.id)
             
             # Check required fields (only if is_complete)
-            if is_complete and question.required and not self._has_value(answer_value):
+            if (
+                is_complete
+                and question.required
+                and question.id in visible_question_ids
+                and not self._has_value(answer_value)
+            ):
                 errors.append(FieldValidationError(
                     question_id=question.id,
                     field="value",
@@ -106,6 +112,74 @@ class ValidationService:
             errors=errors,
             warnings=warnings
         )
+
+    def _get_visible_question_ids(
+        self,
+        form: Form,
+        answer_map: Dict[int, Dict[str, Any]]
+    ) -> set[int]:
+        """Determine which questions are visible based on conditional rules."""
+        visible = {q.id for q in form.questions}
+        if not getattr(form, "conditional_rules", None):
+            return visible
+        
+        for rule in form.conditional_rules:
+            trigger_answer = answer_map.get(rule.trigger_question_id, {})
+            condition_met = self._is_condition_met(trigger_answer, rule.condition_type, rule.condition_value)
+            
+            if condition_met:
+                if rule.action == "show":
+                    visible.add(rule.target_question_id)
+                elif rule.action == "hide":
+                    visible.discard(rule.target_question_id)
+            else:
+                if rule.action == "show":
+                    visible.discard(rule.target_question_id)
+        
+        return visible
+
+    def _is_condition_met(
+        self,
+        answer_value: Dict[str, Any],
+        condition_type: str,
+        condition_value: Optional[str]
+    ) -> bool:
+        """Evaluate conditional logic against an answer value."""
+        if not isinstance(answer_value, dict):
+            return False
+        
+        text = answer_value.get("text")
+        number = answer_value.get("number")
+        choices = answer_value.get("choices") or []
+        
+        if condition_type == "equals":
+            if isinstance(text, str) and text == condition_value:
+                return True
+            if number is not None and str(number) == condition_value:
+                return True
+            if choices and condition_value in choices:
+                return True
+            return False
+        
+        if condition_type == "not_equals":
+            if isinstance(text, str):
+                return text != condition_value
+            if number is not None:
+                return str(number) != condition_value
+            if choices:
+                return condition_value not in choices
+            return True
+        
+        if condition_type == "contains":
+            return isinstance(text, str) and condition_value is not None and condition_value in text
+        
+        if condition_type == "is_not_empty":
+            return self._has_value(answer_value)
+        
+        if condition_type == "is_empty":
+            return not self._has_value(answer_value)
+        
+        return False
     
     def validate_field(
         self, 
@@ -318,13 +392,16 @@ class ValidationService:
             return False
         
         # Check all possible value fields
-        text = answer_value.get('text', '').strip()
+        raw_text = answer_value.get('text')
+        text = raw_text.strip() if isinstance(raw_text, str) else ""
         number = answer_value.get('number')
         date = answer_value.get('date')
-        choices = answer_value.get('choices', [])
+        choices = answer_value.get('choices') or []
         rating = answer_value.get('rating')
+        files = answer_value.get('files') or []
+        file_url = answer_value.get('file_url')
         
-        return bool(text or number is not None or date or choices or rating is not None)
+        return bool(text or number is not None or date or choices or rating is not None or files or file_url)
     
     def generate_validation_token(
         self, 

app/system.ipynb

@@ -1,950 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "c74bd6dd",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "c:\\Users\\arsla\\AppData\\Local\\Programs\\Python\\Python313\\Lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
-      "  from .autonotebook import tqdm as notebook_tqdm\n"
-     ]
-    }
-   ],
-   "source": [
-    "import dspy\n",
-    "import pandas as pd"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "40c180c8",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "\n",
-    "\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "360310bc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# styling_instructions\n",
-    "\n",
-    "dspy.configure(lm=dspy.LM(\"openai/gpt-4o-mini\", max_tokens=4500 ))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "e9920b70",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "\n",
-    "class plan_to_d3(dspy.Signature):\n",
-    "    \"\"\" \"\"\"\n",
-    "    plan = dspy.InputField(desc=\"Planner instructions around what they want to visualize\")\n",
-    "    dataset_context = dspy.InputField(desc=\"Information about the dataset, columns, stats\")\n",
-    "    styling_instructions = dspy.InputField(desc=\"Styling instructions for the charts / dashboard\")\n",
-    "    d3_js_code = dspy.OutputField(desc=\"d3.js code that the js frontend can run to visualize\")\n",
-    "\n",
-    "\n",
-    "# query_enhancer = dspy.Predict(\"original_query, dataset_context ->chart_categories:List[Literal['line_charts','bar_charts','histograms','pie_charts','tabular_and_generic_charts','no_chart']]\")\n",
-    "\n",
-    "plan_to_d3_sign = dspy.Predict(plan_to_d3)\n",
-    "\n",
-    "styling_instructions = [\n",
-    "    {\n",
-    "        \"category\": \"line_charts\",\n",
-    "        \"description\": \"Used to visualize trends and changes over time, often with multiple series.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",  # replaces 'template'\n",
-    "            \"axes\": {\n",
-    "                \"stroke_width\": 0.2,\n",
-    "                \"grid_stroke_width\": 1,\n",
-    "                \"grid_color\": \"#e0e0e0\"\n",
-    "            },\n",
-    "            \"title\": {\n",
-    "                \"bold_html\": True,\n",
-    "                \"include\": True\n",
-    "            },\n",
-    "            \"colors\": \"use color scale (d3.schemeCategory10 or interpolated if >10 lines)\",\n",
-    "            \"annotations\": [\"min\", \"max\"],\n",
-    "            \"number_format\": {\n",
-    "                \"apply_k_m\": True,\n",
-    "                \"thresholds\": {\"K\": 1000, \"M\": 100000},\n",
-    "                \"percentage_decimals\": 2,\n",
-    "                \"percentage_sign\": True\n",
-    "            },\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800, \"margin\": {\"top\": 50, \"right\": 50, \"bottom\": 50, \"left\": 60}}\n",
-    "        }\n",
-    "    },\n",
-    "    {\n",
-    "        \"category\": \"bar_charts\",\n",
-    "        \"description\": \"Useful for comparing discrete categories or groups with bars representing values.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",\n",
-    "            \"axes\": {\n",
-    "                \"stroke_width\": 0.2,\n",
-    "                \"grid_stroke_width\": 1\n",
-    "            },\n",
-    "            \"title\": {\"bold_html\": True, \"include\": True},\n",
-    "            \"annotations\": [\"bar values\"],\n",
-    "            \"bar_style\": {\"padding\": 0.1, \"corner_radius\": 3},\n",
-    "            \"number_format\": {\n",
-    "                \"apply_k_m\": True,\n",
-    "                \"thresholds\": {\"K\": 1000, \"M\": 100000},\n",
-    "                \"percentage_decimals\": 2,\n",
-    "                \"percentage_sign\": True\n",
-    "            },\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800}\n",
-    "        }\n",
-    "    },\n",
-    "    {\n",
-    "        \"category\": \"histograms\",\n",
-    "        \"description\": \"Display the distribution of a data set, useful for returns or frequency distributions.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",\n",
-    "            \"bin_count\": 50,\n",
-    "            \"axes\": {\"stroke_width\": 0.2, \"grid_stroke_width\": 1},\n",
-    "            \"title\": {\"bold_html\": True, \"include\": True},\n",
-    "            \"annotations\": [\"x values\"],\n",
-    "            \"bar_style\": {\"opacity\": 0.8, \"corner_radius\": 2},\n",
-    "            \"number_format\": {\n",
-    "                \"apply_k_m\": True,\n",
-    "                \"thresholds\": {\"K\": 1000, \"M\": 100000},\n",
-    "                \"percentage_decimals\": 2,\n",
-    "                \"percentage_sign\": True\n",
-    "            },\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800}\n",
-    "        }\n",
-    "    },\n",
-    "    {\n",
-    "        \"category\": \"pie_charts\",\n",
-    "        \"description\": \"Show composition or parts of a whole with slices representing categories.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",\n",
-    "            \"top_categories_to_show\": 10,\n",
-    "            \"bundle_rest_as\": \"Others\",\n",
-    "            \"title\": {\"bold_html\": True, \"include\": True},\n",
-    "            \"pie_style\": {\n",
-    "                \"inner_radius_ratio\": 0.0,  # 0 for pie, >0 for donut\n",
-    "                \"pad_angle\": 0.02\n",
-    "            },\n",
-    "            \"annotations\": [\"percentage labels\"],\n",
-    "            \"color_scheme\": \"d3.schemeCategory10\",\n",
-    "            \"number_format\": {\n",
-    "                \"apply_k_m\": True,\n",
-    "                \"thresholds\": {\"K\": 1000, \"M\": 100000},\n",
-    "                \"percentage_decimals\": 2,\n",
-    "                \"percentage_sign\": True\n",
-    "            },\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800}\n",
-    "        }\n",
-    "    },\n",
-    "    {\n",
-    "        \"category\": \"tabular_and_generic_charts\",\n",
-    "        \"description\": \"Applies to charts where number formatting needs flexibility, including mixed or raw data.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",\n",
-    "            \"axes\": {\"stroke_width\": 0.2, \"grid_stroke_width\": 1},\n",
-    "            \"title\": {\"bold_html\": True, \"include\": True},\n",
-    "            \"annotations\": [\"x values\"],\n",
-    "            \"number_format\": {\n",
-    "                \"apply_k_m\": True,\n",
-    "                \"thresholds\": {\"K\": 1000, \"M\": 100000},\n",
-    "                \"exclude_if_commas_present\": True,\n",
-    "                \"exclude_if_not_numeric\": True,\n",
-    "                \"percentage_decimals\": 2,\n",
-    "                \"percentage_sign\": True\n",
-    "            },\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800}\n",
-    "        }\n",
-    "    },\n",
-    "    {\n",
-    "        \"category\": \"heat_maps\",\n",
-    "        \"description\": \"Show data density or intensity using color scales on a matrix or grid.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",\n",
-    "            \"axes\": {\n",
-    "                \"stroke_color\": \"black\",\n",
-    "                \"stroke_width\": 0.2,\n",
-    "                \"grid_stroke_width\": 1\n",
-    "            },\n",
-    "            \"color_scale\": \"d3.interpolateViridis\",\n",
-    "            \"title\": {\"bold_html\": True, \"include\": True},\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800}\n",
-    "        }\n",
-    "    },\n",
-    "    {\n",
-    "        \"category\": \"histogram_distribution\",\n",
-    "        \"description\": \"Specialized histogram for return distributions with opacity control.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",\n",
-    "            \"bar_style\": {\"opacity\": 0.75},\n",
-    "            \"axes\": {\n",
-    "                \"grid_stroke_width\": 1\n",
-    "            },\n",
-    "            \"title\": {\"bold_html\": True, \"include\": True},\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800}\n",
-    "        }\n",
-    "    }\n",
-    "]\n",
-    "\n",
-    "# styling_instructions = [str(chart_dict).replace('\\n','') for chart_dict in styling_instructions]\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "d859fae6",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "\n",
-    "class plan_to_d3(dspy.Signature):\n",
-    "    \"\"\" \"\"\"\n",
-    "    plan = dspy.InputField(desc=\"Planner instructions around what they want to visualize\")\n",
-    "    dataset_context = dspy.InputField(desc=\"Information about the dataset, columns, stats\")\n",
-    "    styling_instructions = dspy.InputField(desc=\"Styling instructions for the charts / dashboard\")\n",
-    "    d3_js_code = dspy.OutputField(desc=\"d3.js code that the js frontend can run to visualize\")\n",
-    "\n",
-    "\n",
-    "# query_enhancer = dspy.Predict(\"original_query, dataset_context ->chart_categories:List[Literal['line_charts','bar_charts','histograms','pie_charts','tabular_and_generic_charts','no_chart']]\")\n",
-    "\n",
-    "plan_to_d3_sign = dspy.Predict(plan_to_d3)\n",
-    "\n",
-    "styling_instructions = [\n",
-    "    {\n",
-    "        \"category\": \"line_charts\",\n",
-    "        \"description\": \"Used to visualize trends and changes over time, often with multiple series.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",  # replaces 'template'\n",
-    "            \"axes\": {\n",
-    "                \"stroke_width\": 0.2,\n",
-    "                \"grid_stroke_width\": 1,\n",
-    "                \"grid_color\": \"#e0e0e0\"\n",
-    "            },\n",
-    "            \"title\": {\n",
-    "                \"bold_html\": True,\n",
-    "                \"include\": True\n",
-    "            },\n",
-    "            \"colors\": \"use color scale (d3.schemeCategory10 or interpolated if >10 lines)\",\n",
-    "            \"annotations\": [\"min\", \"max\"],\n",
-    "            \"number_format\": {\n",
-    "                \"apply_k_m\": True,\n",
-    "                \"thresholds\": {\"K\": 1000, \"M\": 100000},\n",
-    "                \"percentage_decimals\": 2,\n",
-    "                \"percentage_sign\": True\n",
-    "            },\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800, \"margin\": {\"top\": 50, \"right\": 50, \"bottom\": 50, \"left\": 60}}\n",
-    "        }\n",
-    "    },\n",
-    "    {\n",
-    "        \"category\": \"bar_charts\",\n",
-    "        \"description\": \"Useful for comparing discrete categories or groups with bars representing values.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",\n",
-    "            \"axes\": {\n",
-    "                \"stroke_width\": 0.2,\n",
-    "                \"grid_stroke_width\": 1\n",
-    "            },\n",
-    "            \"title\": {\"bold_html\": True, \"include\": True},\n",
-    "            \"annotations\": [\"bar values\"],\n",
-    "            \"bar_style\": {\"padding\": 0.1, \"corner_radius\": 3},\n",
-    "            \"number_format\": {\n",
-    "                \"apply_k_m\": True,\n",
-    "                \"thresholds\": {\"K\": 1000, \"M\": 100000},\n",
-    "                \"percentage_decimals\": 2,\n",
-    "                \"percentage_sign\": True\n",
-    "            },\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800}\n",
-    "        }\n",
-    "    },\n",
-    "    {\n",
-    "        \"category\": \"histograms\",\n",
-    "        \"description\": \"Display the distribution of a data set, useful for returns or frequency distributions.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",\n",
-    "            \"bin_count\": 50,\n",
-    "            \"axes\": {\"stroke_width\": 0.2, \"grid_stroke_width\": 1},\n",
-    "            \"title\": {\"bold_html\": True, \"include\": True},\n",
-    "            \"annotations\": [\"x values\"],\n",
-    "            \"bar_style\": {\"opacity\": 0.8, \"corner_radius\": 2},\n",
-    "            \"number_format\": {\n",
-    "                \"apply_k_m\": True,\n",
-    "                \"thresholds\": {\"K\": 1000, \"M\": 100000},\n",
-    "                \"percentage_decimals\": 2,\n",
-    "                \"percentage_sign\": True\n",
-    "            },\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800}\n",
-    "        }\n",
-    "    },\n",
-    "    {\n",
-    "        \"category\": \"pie_charts\",\n",
-    "        \"description\": \"Show composition or parts of a whole with slices representing categories.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",\n",
-    "            \"top_categories_to_show\": 10,\n",
-    "            \"bundle_rest_as\": \"Others\",\n",
-    "            \"title\": {\"bold_html\": True, \"include\": True},\n",
-    "            \"pie_style\": {\n",
-    "                \"inner_radius_ratio\": 0.0,  # 0 for pie, >0 for donut\n",
-    "                \"pad_angle\": 0.02\n",
-    "            },\n",
-    "            \"annotations\": [\"percentage labels\"],\n",
-    "            \"color_scheme\": \"d3.schemeCategory10\",\n",
-    "            \"number_format\": {\n",
-    "                \"apply_k_m\": True,\n",
-    "                \"thresholds\": {\"K\": 1000, \"M\": 100000},\n",
-    "                \"percentage_decimals\": 2,\n",
-    "                \"percentage_sign\": True\n",
-    "            },\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800}\n",
-    "        }\n",
-    "    },\n",
-    "    {\n",
-    "        \"category\": \"tabular_and_generic_charts\",\n",
-    "        \"description\": \"Applies to charts where number formatting needs flexibility, including mixed or raw data.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",\n",
-    "            \"axes\": {\"stroke_width\": 0.2, \"grid_stroke_width\": 1},\n",
-    "            \"title\": {\"bold_html\": True, \"include\": True},\n",
-    "            \"annotations\": [\"x values\"],\n",
-    "            \"number_format\": {\n",
-    "                \"apply_k_m\": True,\n",
-    "                \"thresholds\": {\"K\": 1000, \"M\": 100000},\n",
-    "                \"exclude_if_commas_present\": True,\n",
-    "                \"exclude_if_not_numeric\": True,\n",
-    "                \"percentage_decimals\": 2,\n",
-    "                \"percentage_sign\": True\n",
-    "            },\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800}\n",
-    "        }\n",
-    "    },\n",
-    "    {\n",
-    "        \"category\": \"heat_maps\",\n",
-    "        \"description\": \"Show data density or intensity using color scales on a matrix or grid.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",\n",
-    "            \"axes\": {\n",
-    "                \"stroke_color\": \"black\",\n",
-    "                \"stroke_width\": 0.2,\n",
-    "                \"grid_stroke_width\": 1\n",
-    "            },\n",
-    "            \"color_scale\": \"d3.interpolateViridis\",\n",
-    "            \"title\": {\"bold_html\": True, \"include\": True},\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800}\n",
-    "        }\n",
-    "    },\n",
-    "    {\n",
-    "        \"category\": \"histogram_distribution\",\n",
-    "        \"description\": \"Specialized histogram for return distributions with opacity control.\",\n",
-    "        \"styling\": {\n",
-    "            \"theme\": \"light\",\n",
-    "            \"bar_style\": {\"opacity\": 0.75},\n",
-    "            \"axes\": {\n",
-    "                \"grid_stroke_width\": 1\n",
-    "            },\n",
-    "            \"title\": {\"bold_html\": True, \"include\": True},\n",
-    "            \"svg_defaults\": {\"height\": 600, \"width\": 800}\n",
-    "        }\n",
-    "    }\n",
-    "]\n",
-    "\n",
-    "# styling_instructions = [str(chart_dict).replace('\\n','') for chart_dict in styling_instructions]\n",
-    "\n",
-    "\n",
-    "class GenerateVisualizationPlan(dspy.Signature):\n",
-    "    \"\"\"\n",
-    "    Generate a visualization or analytical plan from a natural language query.\n",
-    "    The plan should include the reasoning steps and the names of charts\n",
-    "    (chosen from known chart types).\n",
-    "    \"\"\"\n",
-    "\n",
-    "    query = dspy.InputField(\n",
-    "        desc=\"User's natural language query or instruction, e.g., 'Compare monthly revenue trends across regions.'\"\n",
-    "    )\n",
-    "    dataset_context = dspy.InputField(desc=\"Dataset information, columns, types etc\")\n",
-    "    plan = dspy.OutputField(\n",
-    "        desc=(\n",
-    "            \"Step-by-step plan describing what to compute or visualize, \"\n",
-    "            \"including which chart types to use. \"\n",
-    "            \"Chart names must be chosen from: \"\n",
-    "            \"[line_charts, bar_charts, histograms, pie_charts, \"\n",
-    "            \"tabular_and_generic_charts, heat_maps, histogram_distribution].\"\n",
-    "        )\n",
-    "    )\n",
-    "\n",
-    "    relevant_query = dspy.OutputField(\n",
-    "        desc=\"Boolean flag — True if the query is not about visualizations,dashboarding, design, data analysis etc.\",\n",
-    "        type=bool\n",
-    "    )\n",
-    "\n",
-    "\n",
-    "fail = \"\"\"// D3.js: No visualization generated.\n",
-    "// Reason: The query was not about analysis/visualization or the data is invalid.\n",
-    "d3.select(\"body\")\n",
-    "  .append(\"div\")\n",
-    "  .style(\"color\", \"red\")\n",
-    "  .style(\"font-size\", \"1.2em\")\n",
-    "  .style(\"margin\", \"2em\")\n",
-    "  .text(\"No visualization generated: The query was not about analysis/visualization or the data is invalid.\");\"\"\"\n",
-    "\n",
-    "class d3_module(dspy.Module):\n",
-    "    def __init__(self, styling_instructions, fail):\n",
-    "        self.styling_instructions = styling_instructions\n",
-    "        self.planner = dspy.Predict(GenerateVisualizationPlan)\n",
-    "        self.plan_to_d3_sign = dspy.Predict(plan_to_d3)\n",
-    "        self.fail = fail\n",
-    "    def forward(self,query,dataset_context):\n",
-    "        plan = self.planner(query=query, dataset_context=dataset_context)\n",
-    "\n",
-    "        # print(plan)\n",
-    "        if 'False' in str(plan.relevant_query):\n",
-    "            \n",
-    "            styling = []\n",
-    "            for c in self.styling_instructions:\n",
-    "                if c.get(\"category\") in plan.plan:\n",
-    "                    styling.append(c)\n",
-    "            d3_code = self.plan_to_d3_sign(plan=plan.plan, dataset_context = dataset_context, styling_instructions = str(styling) )\n",
-    "\n",
-    "            return d3_code.d3_js_code\n",
-    "        else:\n",
-    "            return self.fail"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "31f9133c",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class d3_module(dspy.Module):\n",
-    "    def __init__(self, styling_instructions, fail):\n",
-    "        self.styling_instructions = styling_instructions\n",
-    "        self.planner = dspy.Predict(GenerateVisualizationPlan)\n",
-    "        self.plan_to_d3_sign = dspy.Predict(plan_to_d3)\n",
-    "        self.fail = fail\n",
-    "    def forward(self,query,dataset_context):\n",
-    "        plan = self.planner(query=query, dataset_context=dataset_context)\n",
-    "\n",
-    "        # print(plan)\n",
-    "        if 'False' in str(plan.relevant_query):\n",
-    "            \n",
-    "            styling = []\n",
-    "            for c in self.styling_instructions:\n",
-    "                if c.get(\"category\") in plan.plan:\n",
-    "                    styling.append(c)\n",
-    "            d3_code = self.plan_to_d3_sign(plan=plan.plan, dataset_context = dataset_context, styling_instructions = str(styling) )\n",
-    "\n",
-    "            return d3_code.d3_js_code\n",
-    "        else:\n",
-    "            return self.fail\n",
-    "            "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 43,
-   "id": "646fbe11",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import pandas as pd\n",
-    "import numpy as np\n",
-    "\n",
-    "# Create a more realistic fake housing dataset DataFrame\n",
-    "np.random.seed(42)\n",
-    "n_samples = 20\n",
-    "\n",
-    "# Bedrooms: Most homes have 2-4 bedrooms, with fewer at the extremes\n",
-    "bedrooms = np.random.choice([1, 2, 3, 4, 5], size=n_samples, p=[0.05, 0.25, 0.45, 0.2, 0.05])\n",
-    "\n",
-    "# Bathrooms: Correlated with bedrooms, but with some noise\n",
-    "bathrooms = np.clip(\n",
-    "    np.random.normal(loc=bedrooms - 0.5 + np.random.uniform(0, 1, n_samples), scale=0.3), 1, 4\n",
-    ")\n",
-    "bathrooms = np.round(bathrooms, 1)\n",
-    "\n",
-    "# Living area: log-normal for positive skew, correlated with bedrooms\n",
-    "sqft_living = np.random.lognormal(mean=7.5 + 0.15 * bedrooms, sigma=0.2, size=n_samples).astype(int)\n",
-    "\n",
-    "# Lot size: log-normal, larger than living area, more variance\n",
-    "sqft_lot = np.random.lognormal(mean=8.5 + 0.1 * bedrooms, sigma=0.35, size=n_samples).astype(int)\n",
-    "\n",
-    "# Floors: Most homes have 1 or 2 floors, rarely 3\n",
-    "floors = np.random.choice([1.0, 1.5, 2.0, 2.5, 3.0], size=n_samples, p=[0.4, 0.15, 0.35, 0.07, 0.03])\n",
-    "\n",
-    "# Waterfront: Only a small fraction are waterfront\n",
-    "waterfront = np.random.choice([True, False], size=n_samples, p=[0.1, 0.9])\n",
-    "\n",
-    "# View: Most homes have 0-1, few have 2-4\n",
-    "view = np.random.choice([0, 1, 2, 3, 4], size=n_samples, p=[0.5, 0.3, 0.1, 0.07, 0.03])\n",
-    "\n",
-    "# Condition: Most homes are in good condition (3-4)\n",
-    "condition = np.random.choice([1, 2, 3, 4, 5], size=n_samples, p=[0.05, 0.15, 0.5, 0.25, 0.05])\n",
-    "\n",
-    "# Grade: Most homes are 6-10, few at extremes\n",
-    "grade = np.random.choice(range(1, 14), size=n_samples, p=[0.01, 0.01, 0.03, 0.05, 0.08, 0.12, 0.18, 0.18, 0.15, 0.10, 0.05, 0.02, 0.02])\n",
-    "\n",
-    "# Year built: Most homes built after 1970, some older\n",
-    "yr_built = np.random.choice(\n",
-    "    np.concatenate([np.arange(1950, 1980), np.arange(1980, 2000), np.arange(2000, 2023)]),\n",
-    "    size=n_samples,\n",
-    "    p=np.concatenate([\n",
-    "        np.repeat(0.01, 30),   # 1950-1979\n",
-    "        np.repeat(0.025, 20),  # 1980-1999\n",
-    "        np.repeat(0.025, 23)   # 2000-2022\n",
-    "    ]) / (0.01*30 + 0.025*43)\n",
-    ")\n",
-    "\n",
-    "# Zipcode: Weighted to favor some zipcodes\n",
-    "zipcodes = ['98101', '98102', '98103', '98104', '98105']\n",
-    "zipcode = np.random.choice(zipcodes, size=n_samples, p=[0.2, 0.15, 0.35, 0.15, 0.15])\n",
-    "\n",
-    "# Latitude and longitude: Centered around Seattle, with small noise\n",
-    "lat = np.round(np.random.normal(loc=47.62, scale=0.04, size=n_samples), 6)\n",
-    "long = np.round(np.random.normal(loc=-122.32, scale=0.04, size=n_samples), 6)\n",
-    "\n",
-    "# Price: Correlated with living area, grade, and waterfront, plus noise\n",
-    "base_price = 100000 + sqft_living * 350 + sqft_lot * 2\n",
-    "grade_adj = (grade - 7) * 20000\n",
-    "waterfront_adj = np.where(waterfront, 150000, 0)\n",
-    "view_adj = view * 10000\n",
-    "price_noise = np.random.normal(0, 40000, n_samples)\n",
-    "price = base_price + grade_adj + waterfront_adj + view_adj + price_noise\n",
-    "price = np.round(price, -3)\n",
-    "\n",
-    "housing_df = pd.DataFrame({\n",
-    "    \"price\": price,\n",
-    "    \"bedrooms\": bedrooms,\n",
-    "    \"bathrooms\": bathrooms,\n",
-    "    \"sqft_living\": sqft_living,\n",
-    "    \"sqft_lot\": sqft_lot,\n",
-    "    \"floors\": floors,\n",
-    "    \"waterfront\": waterfront,\n",
-    "    \"view\": view,\n",
-    "    \"condition\": condition,\n",
-    "    \"grade\": grade,\n",
-    "    \"yr_built\": yr_built,\n",
-    "    \"zipcode\": zipcode,\n",
-    "    \"lat\": lat,\n",
-    "    \"long\": long\n",
-    "})\n",
-    "\n",
-    "housing_df.to_csv(\"housing_sample.csv\")\n",
-    "\n",
-    "\n",
-    "housing_dataset_context = \"\"\"\n",
-    "Dataset Description:\n",
-    "This dataset contains information about residential houses, including their sale prices and various features that may influence price. The data is commonly used for housing market analysis and price prediction.\n",
-    "\n",
-    "Columns:\n",
-    "- price (float): Sale price of the house in USD.\n",
-    "- bedrooms (int): Number of bedrooms in the house.\n",
-    "- bathrooms (float): Number of bathrooms in the house.\n",
-    "- sqft_living (int): Living area in square feet.\n",
-    "- sqft_lot (int): Lot size in square feet.\n",
-    "- floors (float): Number of floors in the house.\n",
-    "- waterfront (bool): Whether the house has a waterfront view (True/False).\n",
-    "- view (int): Quality of the view from the house, rated from 0 (none) to 4 (excellent).\n",
-    "- condition (int): Condition of the house, rated from 1 (poor) to 5 (excellent).\n",
-    "- grade (int): Overall grade of the house, from 1 (low) to 13 (high).\n",
-    "- yr_built (int): Year the house was built.\n",
-    "- zipcode (str): Postal code where the house is located.\n",
-    "- lat (float): Latitude coordinate of the house.\n",
-    "- long (float): Longitude coordinate of the house.\n",
-    "\"\"\"\n",
-    "\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "44036d01",
-   "metadata": {},
-   "source": []
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 40,
-   "id": "7f2e778d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "predictor = d3_module(styling_instructions=styling_instructions, fail=fail)\n",
-    "\n",
-    "response = predictor(query=\"Build me dashboard for the housing dataset, visualize all the variables\", dataset_context=housing_dataset_context )"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 42,
-   "id": "477d0009",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "```javascript\n",
-      "// D3.js code for visualizing the dataset as per the plan\n",
-      "\n",
-      "// 1. Tabular and Generic Chart for Summary Statistics\n",
-      "function createSummaryTable(data) {\n",
-      "    const summaryStats = d3.select(\"#summary-table\")\n",
-      "        .append(\"table\")\n",
-      "        .attr(\"class\", \"summary-table\");\n",
-      "\n",
-      "    const headers = [\"Variable\", \"Mean\", \"Median\", \"Min\", \"Max\", \"Std Dev\"];\n",
-      "    summaryStats.append(\"thead\").append(\"tr\")\n",
-      "        .selectAll(\"th\")\n",
-      "        .data(headers)\n",
-      "        .enter()\n",
-      "        .append(\"th\")\n",
-      "        .text(d => d);\n",
-      "\n",
-      "    const tbody = summaryStats.append(\"tbody\");\n",
-      "    const numericalVars = [\"price\", \"bedrooms\", \"bathrooms\", \"sqft_living\", \"sqft_lot\", \"floors\", \"view\", \"condition\", \"grade\", \"yr_built\"];\n",
-      "    \n",
-      "    numericalVars.forEach(varName => {\n",
-      "        const stats = calculateStats(data, varName); // Assume calculateStats is a function that returns mean, median, min, max, std dev\n",
-      "        tbody.append(\"tr\")\n",
-      "            .selectAll(\"td\")\n",
-      "            .data([varName, stats.mean, stats.median, stats.min, stats.max, stats.stdDev])\n",
-      "            .enter()\n",
-      "            .append(\"td\")\n",
-      "            .text(d => d);\n",
-      "    });\n",
-      "}\n",
-      "\n",
-      "// 2. Histograms for Sale Price, Bedrooms, Bathrooms, and Living Area\n",
-      "function createHistogram(data, variable) {\n",
-      "    const margin = {top: 20, right: 30, bottom: 30, left: 40},\n",
-      "          width = 800 - margin.left - margin.right,\n",
-      "          height = 600 - margin.top - margin.bottom;\n",
-      "\n",
-      "    const svg = d3.select(`#histogram-${variable}`)\n",
-      "        .append(\"svg\")\n",
-      "        .attr(\"width\", width + margin.left + margin.right)\n",
-      "        .attr(\"height\", height + margin.top + margin.bottom)\n",
-      "        .append(\"g\")\n",
-      "        .attr(\"transform\", `translate(${margin.left},${margin.top})`);\n",
-      "\n",
-      "    const x = d3.scaleLinear()\n",
-      "        .domain(d3.extent(data, d => d[variable]))\n",
-      "        .range([0, width]);\n",
-      "\n",
-      "    const histogram = d3.histogram()\n",
-      "        .value(d => d[variable])\n",
-      "        .domain(x.domain())\n",
-      "        .thresholds(x.ticks(50));\n",
-      "\n",
-      "    const bins = histogram(data);\n",
-      "\n",
-      "    const y = d3.scaleLinear()\n",
-      "        .domain([0, d3.max(bins, d => d.length)])\n",
-      "        .range([height, 0]);\n",
-      "\n",
-      "    svg.selectAll(\"rect\")\n",
-      "        .data(bins)\n",
-      "        .enter().append(\"rect\")\n",
-      "        .attr(\"x\", 1)\n",
-      "        .attr(\"transform\", d => `translate(${x(d.x0)},${y(d.length)})`)\n",
-      "        .attr(\"width\", d => x(d.x1) - x(d.x0) - 1)\n",
-      "        .attr(\"height\", d => height - y(d.length))\n",
-      "        .style(\"fill\", \"steelblue\");\n",
-      "\n",
-      "    svg.append(\"g\")\n",
-      "        .attr(\"class\", \"x axis\")\n",
-      "        .attr(\"transform\", `translate(0,${height})`)\n",
-      "        .call(d3.axisBottom(x));\n",
-      "\n",
-      "    svg.append(\"g\")\n",
-      "        .attr(\"class\", \"y axis\")\n",
-      "        .call(d3.axisLeft(y));\n",
-      "}\n",
-      "\n",
-      "// 3. Bar Charts for Average Sale Price by Bedrooms, Bathrooms, and Floors\n",
-      "function createBarChart(data, variable) {\n",
-      "    const margin = {top: 20, right: 30, bottom: 30, left: 40},\n",
-      "          width = 800 - margin.left - margin.right,\n",
-      "          height = 600 - margin.top - margin.bottom;\n",
-      "\n",
-      "    const svg = d3.select(`#bar-chart-${variable}`)\n",
-      "        .append(\"svg\")\n",
-      "        .attr(\"width\", width + margin.left + margin.right)\n",
-      "        .attr(\"height\", height + margin.top + margin.bottom)\n",
-      "        .append(\"g\")\n",
-      "        .attr(\"transform\", `translate(${margin.left},${margin.top})`);\n",
-      "\n",
-      "    const avgPriceData = d3.nest()\n",
-      "        .key(d => d[variable])\n",
-      "        .rollup(v => d3.mean(v, d => d.price))\n",
-      "        .entries(data);\n",
-      "\n",
-      "    const x = d3.scaleBand()\n",
-      "        .domain(avgPriceData.map(d => d.key))\n",
-      "        .range([0, width])\n",
-      "        .padding(0.1);\n",
-      "\n",
-      "    const y = d3.scaleLinear()\n",
-      "        .domain([0, d3.max(avgPriceData, d => d.value)])\n",
-      "        .range([height, 0]);\n",
-      "\n",
-      "    svg.selectAll(\".bar\")\n",
-      "        .data(avgPriceData)\n",
-      "        .enter().append(\"rect\")\n",
-      "        .attr(\"class\", \"bar\")\n",
-      "        .attr(\"x\", d => x(d.key))\n",
-      "        .attr(\"y\", d => y(d.value))\n",
-      "        .attr(\"width\", x.bandwidth())\n",
-      "        .attr(\"height\", d => height - y(d.value))\n",
-      "        .style(\"fill\", \"steelblue\");\n",
-      "\n",
-      "    svg.append(\"g\")\n",
-      "        .attr(\"class\", \"x axis\")\n",
-      "        .attr(\"transform\", `translate(0,${height})`)\n",
-      "        .call(d3.axisBottom(x));\n",
-      "\n",
-      "    svg.append(\"g\")\n",
-      "        .attr(\"class\", \"y axis\")\n",
-      "        .call(d3.axisLeft(y));\n",
-      "}\n",
-      "\n",
-      "// 4. Pie Charts for Waterfront View Proportions\n",
-      "function createPieChart(data) {\n",
-      "    const width = 800, height = 600, radius = Math.min(width, height) / 2;\n",
-      "\n",
-      "    const svg = d3.select(\"#pie-chart\")\n",
-      "        .append(\"svg\")\n",
-      "        .attr(\"width\", width)\n",
-      "        .attr(\"height\", height)\n",
-      "        .append(\"g\")\n",
-      "        .attr(\"transform\", `translate(${width / 2},${height / 2})`);\n",
-      "\n",
-      "    const pie = d3.pie()\n",
-      "        .value(d => d.count);\n",
-      "\n",
-      "    const dataCount = [\n",
-      "        {label: \"Waterfront\", count: d3.sum(data, d => d.waterfront ? 1 : 0)},\n",
-      "        {label: \"No Waterfront\", count: d3.sum(data, d => !d.waterfront ? 1 : 0)}\n",
-      "    ];\n",
-      "\n",
-      "    const arc = d3.arc()\n",
-      "        .innerRadius(0)\n",
-      "        .outerRadius(radius);\n",
-      "\n",
-      "    const color = d3.scaleOrdinal(d3.schemeCategory10);\n",
-      "\n",
-      "    const arcs = svg.selectAll(\".arc\")\n",
-      "        .data(pie(dataCount))\n",
-      "        .enter().append(\"g\")\n",
-      "        .attr(\"class\", \"arc\");\n",
-      "\n",
-      "    arcs.append(\"path\")\n",
-      "        .attr(\"d\", arc)\n",
-      "        .style(\"fill\", (d, i) => color(i));\n",
-      "\n",
-      "    arcs.append(\"text\")\n",
-      "        .attr(\"transform\", d => `translate(${arc.centroid(d)})`)\n",
-      "        .attr(\"dy\", \".35em\")\n",
-      "        .text(d => d.data.label);\n",
-      "}\n",
-      "\n",
-      "// 5. Heat Map for Correlation\n",
-      "function createHeatMap(data) {\n",
-      "    const margin = {top: 20, right: 30, bottom: 30, left: 40},\n",
-      "          width = 800 - margin.left - margin.right,\n",
-      "          height = 600 - margin.top - margin.bottom;\n",
-      "\n",
-      "    const svg = d3.select(\"#heat-map\")\n",
-      "        .append(\"svg\")\n",
-      "        .attr(\"width\", width + margin.left + margin.right)\n",
-      "        .attr(\"height\", height + margin.top + margin.bottom)\n",
-      "        .append(\"g\")\n",
-      "        .attr(\"transform\", `translate(${margin.left},${margin.top})`);\n",
-      "\n",
-      "    const correlationMatrix = calculateCorrelation(data); // Assume calculateCorrelation returns a matrix of correlations\n",
-      "\n",
-      "    const x = d3.scaleBand()\n",
-      "        .domain(correlationMatrix.columns)\n",
-      "        .range([0, width])\n",
-      "        .padding(0.01);\n",
-      "\n",
-      "    const y = d3.scaleBand()\n",
-      "        .domain(correlationMatrix.rows)\n",
-      "        .range([height, 0])\n",
-      "        .padding(0.01);\n",
-      "\n",
-      "    const colorScale = d3.scaleSequential(d3.interpolateRdYlBu)\n",
-      "        .domain([-1, 1]);\n",
-      "\n",
-      "    svg.selectAll(\"rect\")\n",
-      "        .data(correlationMatrix.data)\n",
-      "        .enter().append(\"rect\")\n",
-      "        .attr(\"x\", d => x(d.column))\n",
-      "        .attr(\"y\", d => y(d.row))\n",
-      "        .attr(\"width\", x.bandwidth())\n",
-      "        .attr(\"height\", y.bandwidth())\n",
-      "        .style(\"fill\", d => colorScale(d.value));\n",
-      "\n",
-      "    svg.append(\"g\")\n",
-      "        .attr(\"class\", \"x axis\")\n",
-      "        .attr(\"transform\", `translate(0,${height})`)\n",
-      "        .call(d3.axisBottom(x));\n",
-      "\n",
-      "    svg.append(\"g\")\n",
-      "        .attr(\"class\", \"y axis\")\n",
-      "        .call(d3.axisLeft(y));\n",
-      "}\n",
-      "\n",
-      "// 6. Line Charts for Average Sale Price by Year Built\n",
-      "function createLineChart(data) {\n",
-      "    const margin = {top: 20, right: 30, bottom: 30, left: 40},\n",
-      "          width = 800 - margin.left - margin.right,\n",
-      "          height = 600 - margin.top - margin.bottom;\n",
-      "\n",
-      "    const svg = d3.select(\"#line-chart\")\n",
-      "        .append(\"svg\")\n",
-      "        .attr(\"width\", width + margin.left + margin.right)\n",
-      "        .attr(\"height\", height + margin.top + margin.bottom)\n",
-      "        .append(\"g\")\n",
-      "        .attr(\"transform\", `translate(${margin.left},${margin.top})`);\n",
-      "\n",
-      "    const avgPriceByYear = d3.nest()\n",
-      "        .key(d => d.yr_built)\n",
-      "        .rollup(v => d3.mean(v, d => d.price))\n",
-      "        .entries(data);\n",
-      "\n",
-      "    const x = d3.scaleLinear()\n",
-      "        .domain(d3.extent(avgPriceByYear, d => d.key))\n",
-      "        .range([0, width]);\n",
-      "\n",
-      "    const y = d3.scaleLinear()\n",
-      "        .domain([0, d3.max(avgPriceByYear, d => d.value)])\n",
-      "        .range([height, 0]);\n",
-      "\n",
-      "    const line = d3.line()\n",
-      "        .x(d => x(d.key))\n",
-      "        .y(d => y(d.value));\n",
-      "\n",
-      "    svg.append(\"path\")\n",
-      "        .datum(avgPriceByYear)\n",
-      "        .attr(\"fill\", \"none\")\n",
-      "        .attr(\"stroke\", \"steelblue\")\n",
-      "        .attr(\"stroke-width\", 1.5)\n",
-      "        .attr(\"d\", line);\n",
-      "\n",
-      "    svg.append(\"g\")\n",
-      "        .attr(\"class\", \"x axis\")\n",
-      "        .attr(\"transform\", `translate(0,${height})`)\n",
-      "        .call(d3.axisBottom(x));\n",
-      "\n",
-      "    svg.append(\"g\")\n",
-      "        .attr(\"class\", \"y axis\")\n",
-      "        .call(d3.axisLeft(y));\n",
-      "}\n",
-      "\n",
-      "// 7. Scatter Plot for sqft_living vs Price\n",
-      "function createScatterPlot(data) {\n",
-      "    const margin = {top: 20, right: 30, bottom: 30, left: 40},\n",
-      "          width = 800 - margin.left - margin.right,\n",
-      "          height = 600 - margin.top - margin.bottom;\n",
-      "\n",
-      "    const svg = d3.select(\"#scatter-plot\")\n",
-      "        .append(\"svg\")\n",
-      "        .attr(\"width\", width + margin.left + margin.right)\n",
-      "        .attr(\"height\", height + margin.top + margin.bottom)\n",
-      "        .append(\"g\")\n",
-      "        .attr(\"transform\", `translate(${margin.left},${margin.top})`);\n",
-      "\n",
-      "    const x = d3.scaleLinear()\n",
-      "        .domain(d3.extent(data, d => d.sqft_living))\n",
-      "        .range([0, width]);\n",
-      "\n",
-      "    const y = d3.scaleLinear()\n",
-      "        .domain(d3.extent(data, d => d.price))\n",
-      "        .range([height, 0]);\n",
-      "\n",
-      "    svg.selectAll(\"circle\")\n",
-      "        .data(data)\n",
-      "        .enter().append(\"circle\")\n",
-      "        .attr(\"cx\", d => x(d.sqft_living))\n",
-      "        .attr(\"cy\", d => y(d.price))\n",
-      "        .attr(\"r\", 3)\n",
-      "        .style(\"fill\", \"steelblue\");\n",
-      "\n",
-      "    svg.append(\"g\")\n",
-      "        .attr(\"class\", \"x axis\")\n",
-      "        .attr(\"transform\", `translate(0,${height})`)\n",
-      "        .call(d3.axisBottom(x));\n",
-      "\n",
-      "    svg.append(\"g\")\n",
-      "        .attr(\"class\", \"y axis\")\n",
-      "        .call(d3.axisLeft(y));\n",
-      "}\n",
-      "\n",
-      "// Call the functions with the dataset\n",
-      "createSummaryTable(dataset);\n",
-      "createHistogram(dataset, 'price');\n",
-      "createHistogram(dataset, 'bedrooms');\n",
-      "createHistogram(dataset, 'bathrooms');\n",
-      "createHistogram(dataset, 'sqft_living');\n",
-      "createBarChart(dataset, 'bedrooms');\n",
-      "createBarChart(dataset, 'bathrooms');\n",
-      "createBarChart(dataset, 'floors');\n",
-      "createPieChart(dataset);\n",
-      "createHeatMap(dataset);\n",
-      "createLineChart(dataset);\n",
-      "createScatterPlot(dataset);\n",
-      "```\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(response)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "df811674",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.13.5"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

requirements.txt

@@ -21,6 +21,7 @@ chardet==5.2.0
 openpyxl==3.1.2
 xlrd==2.0.1
 numpy==2.1.2
+boto3==1.34.162
 python-multipart==0.0.20
 tabulate==0.9.0
 plotly==6.3.1
@@ -29,4 +30,3 @@ reportlab==4.0.7
 geoip2==4.8.0
 httpx==0.27.0
 
-