Spaces:
Running
THAMAN — Data Catalog
AI-Powered PropTech System for Property Valuation Last updated: March 2026
Riyadh Data Sources
| # | Dataset | File | Records | Notes |
|---|---|---|---|---|
| R1 | Real Estate Transactions (2018–2023) | data/raw/quarter_report SI.xlsx |
~6,500 rows | Saudi Open Data baseline; district-level quarterly aggregates |
| R2 | RE Transactions 2024 Q1 | data/raw/ (Saudi Open Data CSV) |
~300 rows | Property transactions by type + district |
| R3 | RE Transactions 2024 Q3 | data/raw/ (Saudi Open Data CSV) |
~300 rows | |
| R4 | RE Transactions 2024 Q4 | data/raw/ (Saudi Open Data CSV) |
~300 rows | |
| R5 | RE Transactions 2025 Q1–Q3 | data/raw/ (Saudi Open Data CSVs) |
~600 rows | Holdout period |
| R6 | Metro Stations | data/raw/metro-stations-in-riyadh-*.geojson |
85 stations | 6 lines; opened 2024 |
| R7 | Bus Stops | data/raw/bus-stops-in-riyadh-*.geojson |
~2,500 stops | Includes BRT stops |
| R8 | Traffic Intersections | data/raw/traffic-intersections-*.geojson |
~8,000 intersections | Major street crossings |
| R9 | Commercial Services | data/raw/commercial-services-*.geojson |
~12,000 POIs | 10 categories (hypermarkets, banks, restaurants…) |
| R10 | Air Quality Stations | data/raw/air-quality-stations-*.geojson |
12 stations | NO₂, SO₂, PM₁₀, O₃ measurements |
| R11 | Air Quality Readings | data/raw/air-quality.csv |
~500 rows | Station-level pollutant means |
| R12 | Mosques | data/raw/riyadh_mosques.csv |
~3,000 | OSM-derived |
| R13 | Malls | data/raw/riyadh_malls.csv |
~80 | OSM-derived |
| R14 | Schools | data/raw/riyadh_schools.csv |
~600 | OSM-derived |
| R15 | Hospitals | data/raw/riyadh_hospitals.csv |
~120 | OSM-derived |
| R16 | Parks | data/raw/riyadh_parks.csv |
~200 | OSM-derived |
| R17 | Entertainment Venues | data/raw/rcrc_entertainment.csv |
~150 | RCRC leisure venues |
| R18 | Rental Listings (SA_Aqar) | data/raw/SA_Aqar.csv |
960 listings | District-level medians: size, bedrooms, age, rent/sqm |
| R19 | Real Estate Price Index | data/raw/real-estate-indices.csv |
~200 rows | REI residential + apartment quarterly (2019–2025) |
| R20 | District Polygons | data/processed/riyadh_district_polygons.geojson |
133 polygons | OSM Overpass admin_level=10; 107/133 enriched |
| R21 | District Centroids | data/processed/district_centroids.csv |
147 centroids | Derived from OSM polygon centroids |
Processed output: data/processed/features_riyadh.csv — 6,910 rows × 87 columns
Pipeline: scripts/riyadh_feature_engineering.py (Polars-native)
Changelog
| Date | File | Change |
|---|---|---|
| 2026-03-08 | feature_engineering.py |
Initial pipeline created — 26 columns: property sales, NTA assignment, KD-tree distances (subway, school, park, hospital), BallTree POI count, crime/noise rates, building age |
| 2026-03-08 | data/raw/overture_places.geojson |
Replaced broken OSM POI export (12,886 records) with Overture Maps via leafmap (425,387 records) |
| 2026-03-08 | data/raw/nta_boundaries.geojson |
Added population_2020 (Census 2020 Decennial) and median_income_nta (ACS 5-year 2020) enriched into NTA file via census tract spatial join |
| 2026-03-08 | data/raw/mta_bus_stops.csv |
Added bus stop locations from MTA GTFS S3 feeds (4 boroughs, 9,747 stops); added dist_bus_m feature |
| 2026-03-08 | feature_engineering.py |
Added grouped imputation for gross_square_feet, land_square_feet, residential_units (35.8% null → 0% null) using median by bldgclass + borough |
| 2026-03-08 | feature_engineering.py |
Added renovated_since_2018 and years_since_renovation from DOB A1/A2 permits via NYC Open Data |
| 2026-03-08 | feature_engineering.py |
Added mortgage_rate_30yr from FRED MORTGAGE30US; joined to sales via merge_asof on sale date |
| 2026-03-08 | feature_engineering.py |
Added is_landmark and is_historic_district from LPC dataset (endpoint: ncre-qhxs); matched via BBL |
| 2026-03-08 | feature_engineering.py |
Added dist_waterfront_m from OSM Overpass coastline query (33,507 points) via KD-tree |
| 2026-03-08 | feature_engineering.py |
Added dist_bike_lane_m from OSM Overpass cycleway query (4,155 segments) via KD-tree |
| 2026-03-08 | feature_engineering.py |
Added dist_elem_school_m from NYC elementary school directory (423 schools) via KD-tree |
| 2026-03-08 | feature_engineering.py |
Added dist_express_subway_m and nearest_station_is_express from MTA GTFS Daytime Routes column (380 express stations) |
| 2026-03-08 | feature_engineering.py |
Added livability_complaint_rate from 311 heat/rodent/dirty complaints (383,778 records); normalized per 1k NTA residents |
| 2026-03-08 | feature_engineering.py |
Added borough_income_deviation (NTA median income minus borough median income) |
| 2026-03-08 | feature_engineering.py |
Added sale_year and sale_month for temporal/seasonality signals |
| 2026-03-08 | DATA_CATALOG.md |
Added raw data source entries #14–20 (DOB permits, FRED, elementary schools, 311 livability, LPC landmarks, OSM coastline, OSM bike lanes) |
| 2026-03-08 | DATA_CATALOG.md |
Expanded "What Could Further Improve the Model" from 7 → 15 items with difficulty/notes columns |
| 2026-03-08 | data/processed/features.csv |
Added FAR/zoning fields from PLUTO: residfar, commfar, facilfar, builtfar, maxallwfar, far_utilization (40 → 46 cols). Fix: actual column is commfar not comfar; maxallwfar computed as max(residfar, commfar, facilfar) |
| 2026-03-08 | data/processed/features.csv |
Added building type flags from bldgclass: has_elevator, is_condo, is_multifamily, is_single_fam, is_mixed_use (46 → 51 cols). Fix: fill NaN bldgclass before .str.startswith() |
| 2026-03-08 | data/raw/airbnb_listings.csv |
Downloaded Inside Airbnb NYC December 2025 snapshot (36,261 listings). URL required .csv.gz format + browser User-Agent header; earlier dated snapshot URLs returned 403 |
| 2026-03-08 | data/processed/features.csv |
Added airbnb_count_500m via BallTree — 92.9% of properties have ≥1 Airbnb within 500m; median=18. corr with sale_price: +0.078 (51 → 52 cols) |
| 2026-03-08 | data/processed/features.csv |
Added ACRIS prior sale features: prior_sale_price, prior_sale_date, price_appreciation, years_since_prior_sale, is_flip (52 → 57 cols). Source: ACRIS Master (bnx9-e6tj) + Legals (8h5j-fqxa) joined on document_id. 50 concurrent batches by borough+block in 31s. Match rate: 13.7%; prior_sale_price corr with sale_price: +0.719 |
| 2026-03-08 | data/processed/features.csv |
Added school district quality composite: school_district, district_avg_score, district_school_count (57 → 60 cols). District boundaries from ArcGIS REST (NYC Open Data GeoJSON endpoint returned 400). 272 HS schools across 19 of 33 districts; 56.9% null (districts without HS data) |
| 2026-03-09 | models/xgboost_model.json |
XGBoost trained: 917 trees, R²=0.735, MedAPE=17.8%, MAE=$726,593 on test set. Ridge baseline R²=0.238. Pre-processing: log1p target, winsorize 3 cols, label-encode bldgclass, 80/20 split by borough |
| 2026-03-09 | models/scorer.py |
ThamanScorer class created — loads model, encodes inputs, returns predicted price + confidence range ±17.8% |
| 2026-03-09 | models/shap_importance.png |
SHAP top-20 chart: gross_sqft #1, bldgclass #2, land_sqft #3, longitude #4, latitude #5, school_district #6 |
| 2026-03-09 | models/actual_vs_predicted.png |
Actual vs Predicted scatter (properties < $10M, R²=0.735) |
| 2026-03-09 | models/error_by_borough.png |
% error by borough boxplot |
| 2026-03-08 | data/processed/features.csv |
Full audit + 6 data quality fixes: (1) building_age capped at 200 — 984 outliers fixed; (2) years_since_prior_sale clipped to 0 — 542 negatives fixed; (3) added has_prior_sale binary flag + corrected is_flip logic; (4) district_avg_score imputed with borough→global median — 0% null; (5) numfloors imputed by bldgclass+borough — 0% null; (6) far_utilization nulls filled with 0. Final: 36,203 × 61 cols, all non-ACRIS cols 0% null |
| 2026-03-09 | api/main.py |
FastAPI backend created — POST /predict returns predicted price + SHAP top-10 drivers; POST /batch handles up to 50 properties; CORS enabled for frontend |
| 2026-03-09 | api/spatial.py |
SpatialLookup class — loads 7 spatial datasets at startup; KD-tree for transit/school/park distances; BallTree for Airbnb density; NTA point-in-polygon for crime/income/school-district features |
| 2026-03-09 | api/models.py |
Pydantic schemas — PredictRequest (8 required, 20+ optional fields), PredictResponse (price, range, SHAP drivers, spatial summary) |
| 2026-03-09 | models/scorer.py |
Bug fix: ACRIS defaults changed from 0.0 to np.nan in predict_single — training median imputation now correctly applies (was causing ~100× underpricing) |
| 2026-03-09 | frontend/index.html |
Map UI created — Leaflet.js NYC map + property form + result/SHAP/spatial panels served at /ui |
| 2026-03-09 | frontend/style.css |
Full UI stylesheet — CSS variables, 2-col layout, animated cards, SHAP bars, responsive at 768px |
| 2026-03-09 | frontend/app.js |
Browser JS — map click, borough auto-detect, form validation, fetch /predict, SHAP bar + spatial grid render |
| 2026-03-09 | api/main.py |
Added StaticFiles mount at /ui → frontend/; root / now serves index.html directly; API info moved to /api |
| 2026-03-09 | data/processed/features.csv |
Expanded dataset 5× — from 36,203 rows (2025 only) to 185,092 rows (2022–2026) via scripts/download_more_sales.py. Added 1 new column: assesstot (PLUTO total assessed value) for prior_sale_price imputation (62 cols total) |
| 2026-03-09 | data/processed/features.csv |
Prior sale coverage improved from 13.7% → ~99% using assesstot ratios to estimate prior_sale_price for properties without ACRIS history |
| 2026-03-09 | models/xgboost_model.json |
Retrained XGBoost base learner on 185K rows with Spatial GroupKFold CV (5 folds by NTA) |
| 2026-03-09 | models/thaman_stack.pkl |
Trained stacking ensemble v2.1: XGBoost + LightGBM + CatBoost base learners + Ridge meta-learner. R²=0.6509, MedAPE=20.29% on spatial holdout |
| 2026-03-09 | models/meta.json |
Updated to 71 feature names (added log-dist + target-encoded cols), stack metrics, bldgclass_means, borough_bldg_means, walk_score_scaler |
| 2026-03-09 | api/main.py |
Fixed 500 error on /predict — int(NaN) on school_district spatial lookup. Added _safe_int() and _safe_round() helpers. Updated metrics to v2.1 |
| 2026-03-10 | tests/conftest.py |
Created module-scoped pytest fixture — with TestClient(app) as c triggers FastAPI lifespan (model + spatial data load) for test suite |
| 2026-03-10 | tests/test_api.py |
Fixed 17 test functions to use client fixture; renamed test_root_redirects → test_root_serves_ui (root now returns 200, not redirect) |
| 2026-03-10 | tests/test_scorer.py |
Fixed hardcoded feature counts: 70 → 71 (2 assertions) |
Feature Matrix (Model Input)
File: data/processed/features.csv
Rows: 185,092 properties | Columns: 62
Target variable: sale_price (log-transform required — raw skewness: 54.9)
Sale date range: 2022 – 2026 (NYC Rolling Sales, 4 years)
| Column | Type | Null % | Notes |
|---|---|---|---|
sale_price |
float | 0% | Target. Filter: > $10,000. Log-transform before training |
building_age |
int | 0% | Current year minus yearbuilt |
numfloors |
float | 2.9% | From PLUTO |
bldgclass |
str | 0% | NYC building class code (D4=elevator apt, A1=single family, etc.) |
gross_square_feet |
float | 35.8% | High null rate in Manhattan (condos) — impute with median by bldgclass |
land_square_feet |
float | 35.8% | Same |
residential_units |
float | 35.8% | Same |
dist_subway_m |
float | 0% | KD-tree distance to nearest subway station |
dist_bus_m |
float | 0% | KD-tree distance to nearest bus stop |
dist_express_subway_m |
float | 0% | Distance to nearest express subway station |
nearest_station_is_express |
int (0/1) | 0% | 1 if nearest subway station is an express stop |
dist_elem_school_m |
float | 0% | Distance to nearest elementary school |
dist_waterfront_m |
float | 0% | KD-tree distance to NYC coastline (OSM) |
dist_bike_lane_m |
float | 0% | Distance to nearest bike lane/cycleway |
is_landmark |
int (0/1) | 0% | 1 if property is an NYC Landmark (LPC) |
is_historic_district |
int (0/1) | 0% | 1 if in an LPC Historic District |
renovated_since_2018 |
int (0/1) | 0% | 1 if DOB A1/A2 permit issued since 2018 |
years_since_renovation |
int | 0% | Years since last major permit (999 = never) |
mortgage_rate_30yr |
float | 0% | 30yr fixed mortgage rate at time of sale (FRED) |
sale_year |
int | 0% | Year of sale |
sale_month |
int | 0% | Month of sale (1–12, captures seasonality) |
livability_complaint_rate |
float | 0% | Heat/rodent/dirty complaints per 1k NTA residents |
borough_income_deviation |
float | 0% | NTA median income minus borough median income |
dist_school_m |
float | 0% | KD-tree distance to nearest school |
dist_park_m |
float | 0% | KD-tree distance to nearest park centroid |
dist_hospital_m |
float | 0% | KD-tree distance to nearest health POI (Overture) |
poi_count_500m |
int | 0% | BallTree count of Overture places within 500m |
crime_rate_nta |
float | 0% | Crimes per 1,000 residents in NTA (2022–2024) |
noise_density_nta |
float | 0% | Noise complaints per 1,000 residents in NTA (2022–2024) |
median_income_nta |
float | 0% | Median household income in NTA (ACS 2020) |
population_2020 |
int | 0% | NTA population from 2020 Census |
residfar |
float | 0% | Residential Floor Area Ratio allowed by zoning (PLUTO) |
commfar |
float | 0% | Commercial FAR allowed by zoning (PLUTO) |
facilfar |
float | 0% | Facility FAR allowed by zoning (PLUTO) |
builtfar |
float | 0% | Actual built FAR (PLUTO) |
maxallwfar |
float | 0% | Max allowed FAR = max(residfar, commfar, facilfar) |
far_utilization |
float | 0.3% | builtfar / maxallwfar — how fully the lot is developed (capped at 5) |
has_elevator |
int (0/1) | 0% | 1 if building class is elevator-type (D class, select R class) |
is_condo |
int (0/1) | 0% | 1 if bldgclass starts with R (condo/co-op) |
is_multifamily |
int (0/1) | 0% | 1 if bldgclass starts with D (multifamily elevator) |
is_single_fam |
int (0/1) | 0% | 1 if bldgclass starts with A (single family) |
is_mixed_use |
int (0/1) | 0% | 1 if bldgclass starts with S (mixed use) |
airbnb_count_500m |
int | 0% | Airbnb listings within 500m (Inside Airbnb Dec 2025); median=18, corr=+0.078 |
prior_sale_price |
float | 86.3% | Price of second most recent DEED for same BBL (ACRIS); corr with sale_price: +0.719 |
prior_sale_date |
datetime | 86.3% | Date of prior sale |
price_appreciation |
float | 86.3% | (sale_price - prior_sale_price) / prior_sale_price; clipped to [-1, 10] |
years_since_prior_sale |
float | 86.3% | Years between prior deed and current sale |
is_flip |
int (0/1) | 0% | 1 if prior sale exists AND years_since_prior_sale < 2 |
has_prior_sale |
int (0/1) | 0% | 1 if ACRIS prior deed found for this BBL (13.7% coverage) |
school_district |
int | 0.03% | NYC school district number (1–32) |
district_avg_score |
float | 56.9% | Average graduation rate of HS schools in the district |
district_school_count |
int | 56.9% | Number of high schools in the district |
Raw Data Sources
1. Property Sales
File: data/raw/sales_geocoded.csv
Records: 185,092 filtered (price > $10k) across 2022–2026 | originally 81,305 (2025 only)
Description: NYC property sale transactions across all 5 boroughs. Includes address, sale price, sale date, building class, and structural attributes joined from PLUTO via BBL. Expanded from 1 year (2025) to 4 years (2022–2026) using scripts/download_more_sales.py.
Source: NYC Rolling Calendar Sales — NYC Open Data
2. PLUTO (Building Attributes)
File: data/raw/nyc_pluto_25v4_csv/pluto_25v4.csv
Records: 858,644 parcels | 92 columns
Description: Parcel-level building data for all NYC lots. Provides building age (yearbuilt), floor count, building class, zoning, lot area, and lat/lng coordinates used for geocoding the sales data.
Source: NYC MapPLUTO — NYC Department of City Planning
3. Overture Maps Places (POIs)
File: data/raw/overture_places.geojson
Records: 425,387 places
Description: Comprehensive point-of-interest dataset for NYC covering restaurants, shops, healthcare locations, services, and more. Used for poi_count_500m and dist_hospital_m. Replaced a broken OSM export that had only 12,886 records.
Source: Overture Maps Foundation via leafmap.get_overture_data(type='place', bbox=NYC_BBOX)
4. MTA Subway Stations
File: data/raw/MTA_Subway_Stations_20260308.csv
Records: 496 stations
Description: All NYC subway station locations with GTFS coordinates, line names, and ADA accessibility info. Used to compute dist_subway_m.
Source: MTA Subway Stations — NYC Open Data
5. MTA Bus Stops
File: data/raw/mta_bus_stops.csv
Records: 9,747 stops (Manhattan + Brooklyn + Queens + Bronx)
Description: Bus stop locations from MTA GTFS feeds across all NYC boroughs. Used to compute dist_bus_m for the transit accessibility score.
Source: MTA GTFS Static Feeds — rrgtfsfeeds.s3.amazonaws.com
- Manhattan:
gtfs_m.zip - Brooklyn:
gtfs_b.zip - Queens:
gtfs_q.zip - Bronx:
gtfs_bx.zip
6. Parks
File: data/raw/parks_with_coords.csv
Records: 2,058 parks (with centroids extracted from polygon geometry)
Description: NYC parks properties with WKT polygon geometry. Centroids were extracted using GeoPandas to compute dist_park_m.
Source: Parks Properties — NYC Open Data
7. Schools
File: data/raw/schools.csv
Records: 427 schools
Description: NYC high school locations with quality metrics (graduation rate, attendance rate). Joined from two datasets: school directory (locations) and school quality report (metrics). Used for dist_school_m.
Source (locations): NYC School Locations — NYC Open Data
Source (quality): NYC School Quality Reports — NYC Open Data
8. NYPD Crime Complaint Data
File: data/raw/nypd_crimes.parquet
Records: 1,646,571 complaints (2022–2024)
Description: All NYPD crime complaints with offense type, borough, date, and lat/lng. Used to compute crime_rate_nta (crimes per 1,000 NTA residents).
Source: NYPD Complaint Data Historic — NYC Open Data
9. NYC 311 Noise Complaints
File: data/raw/noise_complaints.parquet
Records: 1,000,000 complaints (2022–2024, filtered to Noise types)
Description: NYC 311 noise service requests with complaint type, date, and lat/lng. Types include Residential, Street/Sidewalk, Commercial, Vehicle. Used to compute noise_density_nta.
Source: 311 Service Requests — NYC Open Data
10. NTA Boundaries + Population + Income
File: data/raw/nta_boundaries.geojson
Records: 262 neighborhoods
Description: 2020 Neighborhood Tabulation Area boundaries enriched with:
population_2020— from 2020 Decennial Census (P1_001N by tract, aggregated to NTA)median_income_nta— from ACS 5-year 2020 (B19013_001E by tract, aggregated to NTA) Used to normalize crime and noise rates per 1,000 residents. Source (boundaries): NYC NTA 2020 — ArcGIS REST API Source (population): US Census 2020 Decennial API Source (income): US Census ACS 5-year 2020 API
11. Road Network
Files: data/raw/road_network/*.graphml (10 files)
Records: Drive + Walk networks for all 5 boroughs
Description: Full road and pedestrian network graphs downloaded via OSMnx from OpenStreetMap. Can be used for accurate network-distance routing (vs straight-line KD-tree approximation) and road type analysis.
| Borough | Drive nodes | Drive edges | Walk nodes | Walk edges |
|---|---|---|---|---|
| Manhattan | 4,628 | 9,916 | 36,631 | 116,868 |
| Brooklyn | 12,224 | 30,350 | 69,677 | 224,888 |
| Queens | 21,491 | 55,972 | 123,096 | 400,104 |
| Bronx | 7,723 | 19,259 | 39,392 | 120,888 |
| Staten Island | 9,209 | 23,464 | 59,469 | 169,418 |
Source: OpenStreetMap via OSMnx
12. Census Tract Population
File: data/raw/census_tract_population.csv
Records: 2,327 NYC tracts
Description: 2020 Decennial Census population (P1_001N) for every census tract in NYC's 5 counties. Used as an intermediate step to compute NTA-level population via spatial join.
Source: US Census Bureau 2020 Decennial API
13. Census Tract Income
File: data/raw/census_tract_income.csv
Records: 2,327 NYC tracts
Description: ACS 5-year 2020 median household income (B19013_001E) by census tract. Used as an intermediate step to compute NTA-level median income via spatial join.
Source: US Census Bureau ACS 5-year 2020 API
14. DOB Renovation Permits
File: data/raw/dob_permits.csv
Records: 2,673 unique BBLs with A1/A2 permits since 2018
Description: NYC Department of Buildings alteration permits (job types A1/A2) filed since 2018. Used to compute renovated_since_2018 (binary flag) and years_since_renovation. BBL constructed from borough + block + lot columns matching PLUTO format.
Source: DOB Permit Issuance — NYC Open Data
15. FRED Mortgage Rates
File: data/raw/mortgage_rates.csv
Records: ~1,500 weekly observations (1971–2026)
Description: Weekly 30-year fixed mortgage rate from the Federal Reserve Economic Data (FRED). Joined to each property sale via merge_asof on sale date. Used as mortgage_rate_30yr feature capturing macro credit conditions at time of sale.
Source: FRED MORTGAGE30US — St. Louis Federal Reserve
16. Elementary Schools
File: data/raw/elementary_schools.csv
Records: 423 elementary schools
Description: NYC public elementary school (K–5/PK) locations filtered from the NYC school directory. Used to compute dist_elem_school_m via KD-tree. Separate from the high school dataset to capture proximity to schools relevant to families with young children.
Source: NYC School Locations — NYC Open Data
17. NYC 311 Livability Complaints
File: data/raw/livability_complaints.parquet
Records: 383,778 complaints (2022–2024)
Description: NYC 311 service requests filtered to livability issue types: heat/hot water, rodent, and dirty conditions. Aggregated to NTA level and normalized per 1,000 residents to compute livability_complaint_rate. Complements noise complaints as a separate QoL dimension.
Source: 311 Service Requests — NYC Open Data
18. LPC Landmarks
File: Used directly from Socrata API (not cached)
Records: 39,385 landmark records → 3,695 unique landmark BBLs
Description: New York City Landmarks Preservation Commission (LPC) designated landmarks and historic district properties. Matched to sales via BBL to compute is_landmark and is_historic_district. Landmarks can affect both value (premium) and renovation flexibility.
Source: LPC Individual Landmarks — NYC Open Data
19. NYC Coastline (OSM Overpass)
File: Used in-memory during pipeline (not saved separately)
Records: 33,507 coastline points
Description: NYC shoreline geometry retrieved via OpenStreetMap Overpass API (natural=coastline within NYC bounding box). Used to compute dist_waterfront_m via KD-tree. Waterfront proximity is a major price driver in NYC, particularly for properties in Battery Park, DUMBO, Red Hook, and Williamsburg.
Source: OpenStreetMap Overpass API — query: natural=coastline within NYC bbox
20. Bike Lanes (OSM Overpass)
File: Used in-memory during pipeline (not saved separately)
Records: 4,155 bike lane segments → coordinate points extracted
Description: NYC bike lane and cycleway geometries from OpenStreetMap via Overpass API. Segment coordinates extracted and used for KD-tree nearest-distance calculation (dist_bike_lane_m). Captures micro-mobility infrastructure as a livability and urban quality indicator.
Source: OpenStreetMap Overpass API — query: highway=cycleway + cycleway tags within NYC bbox
21. Inside Airbnb NYC Listings
File: data/raw/airbnb_listings.csv
Records: 36,261 active listings (December 2025 snapshot)
Description: Short-term rental listings across all 5 boroughs. Used to compute airbnb_count_500m — density of Airbnb units within 500m radius via BallTree. High Airbnb density correlates with tourist-heavy areas and mixed-use neighborhoods. Correlation with sale_price: +0.078.
Source: Inside Airbnb — New York City (free, no key). Note: use .csv.gz URL format + browser User-Agent header; plain CSV URLs return 403.
22. ACRIS Real Property Master
File: Used directly from Socrata API (not cached)
Records: 500,000 most recent DEED transactions (2011–2026)
Description: NYC property deed transfer records from the Automated City Register Information System. Contains document_id, sale date, and sale amount. Does not contain BBL — must be joined with ACRIS Legals (source #23) on document_id to obtain the BBL.
Source: ACRIS Real Property Master — NYC Open Data
23. ACRIS Real Property Legals
File: Used directly from Socrata API (not cached)
Records: 22.5M total; ~24k fetched via 50 batched borough+block queries
Description: Maps each ACRIS document_id to its parcel (borough, block, lot). Used in combination with ACRIS Master to build prior_sale_price, price_appreciation, years_since_prior_sale, and is_flip. Downloaded in 50 concurrent batches (300 blocks per batch) in ~31 seconds.
Source: ACRIS Real Property Legals — NYC Open Data
24. NYC School Districts
File: Used in-memory during pipeline (not saved separately)
Records: 33 school districts
Description: NYC school district boundary polygons. Used for spatial join to assign each property to a district, then joined with school quality metrics to compute district_avg_score and district_school_count. Note: NYC Open Data GeoJSON export endpoint returned 400 — used ArcGIS REST API instead.
Source: NYC School Districts — ArcGIS REST API
Data Quality Notes
Last audit: 2026-03-10 — All 24 raw sources present on disk ✓ | Feature matrix: 185,092 × 62 cols
| Issue | Detail | Status | Action Taken |
|---|---|---|---|
| Price skewness | Raw skewness = 54.86 | ⚠️ Requires transform | Apply log1p(sale_price) before training — log skewness = 0.78 ✓ |
| gross_sqft nulls | 35.8% null (Manhattan condos) | ✅ Fixed | Imputed with median by bldgclass + borough → 0% null |
| NTA outliers | BX1203 noise=2,711/1k, MN0502 crime=460/1k | ⚠️ Requires transform | Winsorize or log-transform at 99th percentile before training |
| Borough encoding | Stored as integers 1–5 | ✅ OK | Map: 1=Manhattan, 2=Bronx, 3=Brooklyn, 4=Queens, 5=Staten Island |
| building_age outlier | 984 properties with age > 200 (yearbuilt=0 in source) | ✅ Fixed | Capped at 200 years |
| years_since_prior_sale negatives | 542 properties with negative values (ACRIS ordering anomaly) | ✅ Fixed | Clipped to 0 minimum |
| is_flip ambiguity | Was 0 for no-prior-sale properties (misleading) | ✅ Fixed | Added has_prior_sale flag; is_flip only set 1 where prior sale exists |
| district_avg_score nulls | 56.9% null (districts with no HS data) | ✅ Fixed | Imputed with borough median → global median fallback → 0% null |
| numfloors nulls | 2.9% null | ✅ Fixed | Imputed with median by bldgclass + borough → 0% null |
| far_utilization nulls | 0.3% null (vacant lots with no FAR) | ✅ Fixed | Filled with 0 (no development) |
| ACRIS prior sale coverage | 86.3% null — window limited to 2011–2026 | ⚠️ Known limitation | Use has_prior_sale as binary feature; model imputes missing with tree splits |
| NTA edge properties | 13 properties fell outside NTA boundary | ✅ Fixed | Filled with borough median for all NTA-level features |
| dist_bus_m outlier | Max 26,807m (Staten Island, no bus service) | ✅ Valid | Real data — Staten Island bus coverage is sparse. Keep as-is |
| dist_express_subway_m outlier | Max 27,592m (Staten Island — no express service) | ✅ Valid | Real data — Staten Island has no express subway. Keep as-is |
Model Readiness Checklist
| Check | Status | Notes |
|---|---|---|
| All raw files on disk | ✅ 19/19 files present | Road network = 10 GraphML files |
| Feature matrix shape | ✅ 185,092 × 62 cols | Expanded to 4 years of sales (2022–2026) |
| Non-ACRIS nulls | ✅ 0% null | All columns except ACRIS prior-sale group fully populated |
| ACRIS prior sale coverage | ✅ ~99% | prior_sale_price coverage improved using assesstot ratios for properties without ACRIS history |
| Target variable | ✅ Ready | Apply log1p() before training (skewness 54.86 → 0.78) |
| Categorical encoding | ✅ Done | bldgclass → target-encoded bldgclass_encoded + borough_bldg_encoded (cross-feature) |
| NTA outliers | ✅ Done | Winsorized at 99th percentile (crime_rate_nta, noise_density_nta, livability_complaint_rate) |
| Train/test split | ✅ Done | Spatial GroupKFold CV (5 folds by NTA) — prevents geographic leakage |
| Model | ✅ Done | Stack v2.1: XGB + LGB + CatBoost + Ridge meta |
What Could Further Improve the Model
| Data | Impact | Difficulty | Notes |
|---|---|---|---|
| Floor level (condo unit floor #) | Very High — strongest predictor for apartments | Very Hard | Not in public data; ACRIS deeds sometimes mention unit/floor but inconsistent |
| View / floor premium (condos) | High | Very Hard | Would require scraping StreetEasy/Zillow — not in public records |
| Interior sqft (condos) | High — 35.8% null in PLUTO for condos | Medium | ACRIS mortgage docs sometimes contain unit sizes; complex to parse |
| Elementary school quality ratings | High — distance added, quality not yet scored | Medium | NYC DOE K-8 quality reports; same long-format as HS dataset; join on dbn |
| ACRIS prior sale — expand window | Medium — only 13.7% matched | Easy | Add more 500k batches: $offset=500000, $offset=1000000 to reach pre-2011 deeds |
| Air quality index by NTA | Medium — pollution negatively affects prices | Medium | EPA AQS API (free key at aqs.epa.gov); param 88101=PM2.5; IDW interpolation from monitors |
| Flood risk score | Medium — FEMA zones suppress coastal prices | Medium | FEMA NFHL API had SSL errors; download GDB locally from msc.fema.gov |
| Foreclosure / lis pendens rate | Medium — financial distress signal | Medium | ACRIS doc_type='LIS PEN'; same pipeline as sources #22–23 |
| Walk Score proxy | Medium — composite pedestrian signal | Easy | Compute from existing features: weighted(dist_subway, dist_bus, poi_count_500m) |
| Shadow / sunlight hours | Low–Medium | Hard | Requires NYC 3D Building Model + solar angle computation (pysolar) |