AuthorBot Cursor commited on
Commit
7698f17
·
1 Parent(s): ecd655e

feat: canonicalize share/short book URLs + deep multi-format price enrichment

Browse files

Resolve a.co/amzn.to/bn.com/itunes families before scrape; wire Thrift/Abe/Bookshop/Kobo deep parsers into presence enrichment; Author UI shows fetched-only chips and Resolved toast.

Co-authored-by: Cursor <cursoragent@cursor.com>

.agent/BDD_SPECS.md CHANGED
@@ -371,6 +371,26 @@ Feature: Book URL Import
371
  And fall back to catalog APIs and any prior cached metadata
372
  And the response should include a human-readable warning instead of a blank 0% match
373
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
374
 
375
  Feature: Platform Presence Scan
376
 
@@ -444,11 +464,27 @@ Feature: Platform List Pricing
444
  And discovery platforms such as goodreads and open_library should have empty format_prices
445
  And the single list_price on each row should equal primary_list_price of the format map
446
 
 
 
 
 
 
 
 
 
 
447
  Scenario: Richer existing format_prices are not wiped by a thinner scan
448
  Given a book with an Amazon listing that has format_prices Hardcover $35 and eBook $14.99
449
  When the same book is re-scanned and the scan only provides a single list_price of $35
450
  Then the Amazon format_prices should still contain both Hardcover and eBook entries
451
 
 
 
 
 
 
 
 
452
  ## Enterprise Features (2026-07)
453
 
454
  Feature: Webhooks
 
371
  And fall back to catalog APIs and any prior cached metadata
372
  And the response should include a human-readable warning instead of a blank 0% match
373
 
374
+ Scenario: Amazon Share short link resolves like a browser
375
+ Given an author pastes https://a.co/d/04YALfQb
376
+ When the author clicks "Fetch Details"
377
+ Then the platform should be amazon (never unknown)
378
+ And the URL should canonicalize to https://www.amazon.com/dp/{ASIN}
379
+ And title, description, about_author, and format_prices should match a clean /dp/ import
380
+ And a soft toast should note Resolved a.co → the canonical product URL
381
+
382
+ Scenario: Barnes and Noble id-only and short-host URLs are supported
383
+ Given an author pastes barnesandnoble.com/w/{id} without a slug or a bn.com short host
384
+ When the author clicks "Fetch Details"
385
+ Then the platform should be barnes_noble
386
+ And identifiers and multi-format prices should resolve when the product page exposes them
387
+
388
+ Scenario: Legacy iTunes Apple Books URL rewrites to books.apple.com
389
+ Given an author pastes itunes.apple.com/{cc}/book/.../id{N}
390
+ When the author clicks "Fetch Details"
391
+ Then the platform should be apple_books
392
+ And the canonical URL should use books.apple.com
393
+
394
 
395
  Feature: Platform Presence Scan
396
 
 
464
  And discovery platforms such as goodreads and open_library should have empty format_prices
465
  And the single list_price on each row should equal primary_list_price of the format map
466
 
467
+ Scenario: Author URL preview enriches every discovered retailer with credentials
468
+ Given an author pastes one supported book URL into Upload Book
469
+ And configured platform credentials are stored in the database
470
+ When the author clicks Fetch Details
471
+ Then the presence scan receives the database and Redis dependencies
472
+ And every verified discovered listing URL is checked for available format prices
473
+ And the preview shows only fetched Hardcover, Paperback, eBook, or Audiobook labels
474
+ And unfetched formats are omitted rather than invented
475
+
476
  Scenario: Richer existing format_prices are not wiped by a thinner scan
477
  Given a book with an Amazon listing that has format_prices Hardcover $35 and eBook $14.99
478
  When the same book is re-scanned and the scan only provides a single list_price of $35
479
  Then the Amazon format_prices should still contain both Hardcover and eBook entries
480
 
481
+ Scenario: Discovered listings use deep multi-format parsers
482
+ Given a presence scan verifies Bookshop, ThriftBooks, AbeBooks, or Kobo listing URLs
483
+ When format prices are enriched
484
+ Then the same deep extractors used on source import should populate format_prices
485
+ And only formats present on the retailer page are stored
486
+ And missing formats are omitted rather than invented
487
+
488
  ## Enterprise Features (2026-07)
489
 
490
  Feature: Webhooks
.agent/BOOK_URL_IMPORT_RULES.md CHANGED
@@ -36,12 +36,28 @@ Code mirror: `LOCKED_URL_IMPORT_PLATFORMS` in `app/services/book_url_scraper.py`
36
 
37
  ## Pipeline (in order — do not skip steps)
38
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
39
  ### Step 1 — Parse the URL (zero HTTP)
40
  | Extract | Pattern |
41
  |---------|---------|
42
- | ASIN | `/dp/{10-char}` or `/gp/product/{10-char}` |
43
  | ISBN | digits in path (AbeBooks, Bookshop, etc.) |
44
- | Platform | domain regex |
 
45
 
46
  Normalize URL: strip `?tag=`, `?ref_=`, canonicalize Amazon to `/dp/{ASIN}`.
47
 
@@ -246,6 +262,14 @@ lives on **`book_platform_listings`** (one row per book × platform).
246
  3. Discovery platforms (Goodreads, Open Library) store empty `format_prices`.
247
  4. Used-print (ThriftBooks, AbeBooks) keep print bindings only; digital
248
  (Apple, Kobo) keep eBook/Audiobook only.
 
 
 
 
 
 
 
 
249
 
250
  **Admin:** Book list → Platforms panel edits URL / ISBN / ASIN / format prices
251
  via `PUT …/books/{id}/platforms/{platform_id}`.
@@ -265,3 +289,22 @@ And genre contains "Kidnapping" or "Crime Fiction"
265
  And isbn field = "B0DHWS3P6T" (ASIN — no ISBN-13 on page)
266
  And confidence = 100%
267
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
 
37
  ## Pipeline (in order — do not skip steps)
38
 
39
+ ### Step 0 — Canonicalize URL (before platform detect)
40
+ Every locked platform has multiple valid URL families for the same book
41
+ (stable product, affiliate/tracking, Share/short/wrapper). Share/short hosts
42
+ are first-class members of that platform — never `"unknown"`.
43
+
44
+ | Rule | Behavior |
45
+ |------|----------|
46
+ | R-URL-001 | Classify by **host-family map** (all 9 platforms + aliases) before scrape |
47
+ | R-URL-002 | Share/short/wrapper hosts (`a.co`, `amzn.to`, `bn.com`, Rakuten wrappers, etc.) MUST follow HTTP redirects like a browser, then rewrite |
48
+ | R-URL-003 | After resolve, rewrite to the platform’s stable product URL (Amazon `/dp/{ASIN}`, BN `/w/…`, Apple `books.apple.com/…/id{N}`, …) |
49
+ | R-URL-004 | Strip tracking params; never drop `ean` / ISBN / product ids |
50
+ | R-URL-005 | Canonicalize runs for pasted import URLs **and** discovered presence listing URLs |
51
+
52
+ Code: `app/services/url_canonicalizer.py` → `canonicalize_book_url`.
53
+
54
  ### Step 1 — Parse the URL (zero HTTP)
55
  | Extract | Pattern |
56
  |---------|---------|
57
+ | ASIN | `/dp/{10-char}` or `/gp/product/{10-char}` (also `/gp/aw/d/`) |
58
  | ISBN | digits in path (AbeBooks, Bookshop, etc.) |
59
+ | BN id | `/w/{slug}/{id}` **or** `/w/{id}` |
60
+ | Platform | host-family map, then domain regex |
61
 
62
  Normalize URL: strip `?tag=`, `?ref_=`, canonicalize Amazon to `/dp/{ASIN}`.
63
 
 
262
  3. Discovery platforms (Goodreads, Open Library) store empty `format_prices`.
263
  4. Used-print (ThriftBooks, AbeBooks) keep print bindings only; digital
264
  (Apple, Kobo) keep eBook/Audiobook only.
265
+ 5. Author **Fetch Details** MUST pass DB + Redis into the presence scan so
266
+ configured retailer/ScrapingBee credentials are available before the scan
267
+ is cached. Every verified discovered listing URL is enriched independently
268
+ via the same deep format parsers used on source import (R-PP-015).
269
+ 6. The Author preview shows only successfully fetched format-price labels.
270
+ Missing formats are omitted; no empty label or invented amount is stored.
271
+ When a share/short URL was resolved, show a soft toast:
272
+ `Resolved {host} → {canonical}`.
273
 
274
  **Admin:** Book list → Platforms panel edits URL / ISBN / ASIN / format prices
275
  via `PUT …/books/{id}/platforms/{platform_id}`.
 
289
  And isbn field = "B0DHWS3P6T" (ASIN — no ISBN-13 on page)
290
  And confidence = 100%
291
  ```
292
+
293
+ ## BDD acceptance (URL families)
294
+ ```
295
+ Given Amazon Share URL https://a.co/d/{code}
296
+ When author clicks Fetch Details
297
+ Then platform = amazon (never unknown)
298
+ And canonical URL is https://www.amazon.com/dp/{ASIN}
299
+ And title / description / about_author / format_prices match the clean /dp/ import
300
+
301
+ Given Barnes & Noble URL …/w/{id} (no slug) or bn.com short host
302
+ When author clicks Fetch Details
303
+ Then platform = barnes_noble
304
+ And identifiers and multi-format prices resolve when the page exposes them
305
+
306
+ Given itunes.apple.com/…/id{N}
307
+ When author clicks Fetch Details
308
+ Then platform = apple_books
309
+ And URL rewrites to books.apple.com
310
+ ```
.agent/FILE_MAP.md CHANGED
@@ -109,6 +109,7 @@
109
  | `app/superadmin/routers/google_oauth_config.py` | SuperAdmin Google OAuth client config |
110
  | `app/services/book_service.py` | Admin book CRUD, covers, URL import, platform presence |
111
  | `app/services/platform_catalog.py` | Canonical format prices + listing view for admin/bot |
 
112
  | `app/services/price_catalog_service.py` | Grounded visitor price answers + platform_offers payload |
113
  | `.agent/PRICE_INTELLIGENCE_RULES.md` | Bot price intelligence rules (R-PI-001–006, format-aware grid) |
114
  | `app/services/admin_analytics_service.py` | Admin dashboard analytics (funnel, heatmap, visitors) |
 
109
  | `app/superadmin/routers/google_oauth_config.py` | SuperAdmin Google OAuth client config |
110
  | `app/services/book_service.py` | Admin book CRUD, covers, URL import, platform presence |
111
  | `app/services/platform_catalog.py` | Canonical format prices + listing view for admin/bot |
112
+ | `app/services/url_canonicalizer.py` | Host-family map + share/short URL canonicalize for 9 platforms |
113
  | `app/services/price_catalog_service.py` | Grounded visitor price answers + platform_offers payload |
114
  | `.agent/PRICE_INTELLIGENCE_RULES.md` | Bot price intelligence rules (R-PI-001–006, format-aware grid) |
115
  | `app/services/admin_analytics_service.py` | Admin dashboard analytics (funnel, heatmap, visitors) |
.agent/PLATFORM_PRICING_RULES.md CHANGED
@@ -112,11 +112,15 @@ Amazon HTML: US-locale headers + cookie (`i18n-prefs=USD`) to avoid PKR/wrong cu
112
  - **R-PP-012:** Discovery platforms (goodreads, open_library, google_books) return `format_prices: {}` from enrichment; their single `list_price` is preserved as-is
113
  - **R-PP-013:** `fetch_listing_price` remains a backward-compatible thin wrapper over `fetch_listing_format_prices` returning only the primary formatted string
114
  - **R-PP-014:** Apple Books enrichment MUST query iTunes `entity=ebook` and `entity=audiobook` so both digital formats can populate `format_prices` when available
 
 
 
115
 
116
  ## 11. File map
117
 
118
  | File | Role |
119
  |------|------|
 
120
  | `app/services/platform_pricing.py` | Fetch, parse, format; `FormatPriceHit`, `fetch_listing_format_prices` |
121
  | `app/services/platform_presence.py` | `_enrich_listing_prices()` orchestration; sets `listing.format_prices` |
122
  | `app/services/platform_api_registry.py` | Credential-tier price fetch |
 
112
  - **R-PP-012:** Discovery platforms (goodreads, open_library, google_books) return `format_prices: {}` from enrichment; their single `list_price` is preserved as-is
113
  - **R-PP-013:** `fetch_listing_price` remains a backward-compatible thin wrapper over `fetch_listing_format_prices` returning only the primary formatted string
114
  - **R-PP-014:** Apple Books enrichment MUST query iTunes `entity=ebook` and `entity=audiobook` so both digital formats can populate `format_prices` when available
115
+ - **R-PP-015:** Presence enrichment MUST call the same deep format parsers as source import — Amazon, BN, ThriftBooks, AbeBooks, Bookshop, Kobo — after one HTML fetch. Never invent missing format keys.
116
+ - **R-PP-016:** Listing URLs MUST be canonicalized (share/short/affiliate → product URL) before price HTML fetch
117
+ - **R-PP-017:** Scan cache key version bumped to `platform_scan:v13:` when canonicalize + deep enrichment ship
118
 
119
  ## 11. File map
120
 
121
  | File | Role |
122
  |------|------|
123
+ | `app/services/url_canonicalizer.py` | Host-family map + `canonicalize_book_url` |
124
  | `app/services/platform_pricing.py` | Fetch, parse, format; `FormatPriceHit`, `fetch_listing_format_prices` |
125
  | `app/services/platform_presence.py` | `_enrich_listing_prices()` orchestration; sets `listing.format_prices` |
126
  | `app/services/platform_api_registry.py` | Credential-tier price fetch |
app/admin/templates/admin.html CHANGED
@@ -666,6 +666,7 @@
666
  .ab-pp-fmt-pair { display: inline-flex; align-items: center; gap: 3px; background: var(--bg2, #f3f4f6); border-radius: 4px; padding: 1px 5px; }
667
  .ab-pp-fmt-name { font-size: 10px; color: var(--muted); }
668
  .ab-pp-fmt-price { font-size: 12px; font-weight: 600; color: var(--text2); }
 
669
  .ab-found-links {
670
  display: flex;
671
  flex-wrap: wrap;
@@ -5983,9 +5984,10 @@ function abRenderPlatformScan(pp) {
5983
  .filter(k => fps[k])
5984
  .map(k => `<span class="ab-pp-fmt-pair"><span class="ab-pp-fmt-name">${esc(k)}</span> <span class="ab-pp-fmt-price">${esc(fps[k])}</span></span>`)
5985
  .join('');
5986
- return parts ? `<div class="ab-pp-price ab-pp-price--fmts">${parts}</div>` : (p.price ? `<div class="ab-pp-price">${esc(p.price)}</div>` : '');
5987
  }
5988
- return p.price ? `<div class="ab-pp-price">${esc(p.price)}</div>` : '';
 
5989
  })();
5990
  const attrs = missing
5991
  ? ` data-pp-platform-id="${esc(p.platform_id)}" data-pp-missing="1" role="button" tabindex="0" aria-label="Not found on ${esc(p.display_name)} — click for publishing help"`
@@ -6142,8 +6144,22 @@ function abRenderPreview(data) {
6142
  document.getElementById('ab-e-bio').value = data.about_author || '';
6143
 
6144
  if (data.warnings && data.warnings.length) {
6145
- abShowAlert('warning', '⚠️ Some data was enriched from APIs: ' + data.warnings.join('; '));
6146
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6147
  abRenderPlatformScan(data.platform_presence);
6148
  abSetState('preview');
6149
  }
@@ -6293,7 +6309,11 @@ async function abLoadSidebar() {
6293
  function abShowAlert(type, msg) {
6294
  const el = document.getElementById('ab-alert');
6295
  if (!el) return;
6296
- const colors = { error:'var(--red-bg);border:1px solid rgba(239,68,68,.25);color:var(--red)', warning:'var(--yellow-bg);border:1px solid rgba(245,158,11,.25);color:#92400e' };
 
 
 
 
6297
  el.style.cssText = `display:flex;align-items:flex-start;gap:10px;padding:11px 14px;border-radius:var(--radius-sm);margin-bottom:14px;font-size:12px;background:${colors[type]||colors.error}`;
6298
  el.textContent = msg;
6299
  }
 
666
  .ab-pp-fmt-pair { display: inline-flex; align-items: center; gap: 3px; background: var(--bg2, #f3f4f6); border-radius: 4px; padding: 1px 5px; }
667
  .ab-pp-fmt-name { font-size: 10px; color: var(--muted); }
668
  .ab-pp-fmt-price { font-size: 12px; font-weight: 600; color: var(--text2); }
669
+ .ab-pp-price--na { font-size: 11px; font-weight: 500; color: var(--muted); font-style: italic; }
670
  .ab-found-links {
671
  display: flex;
672
  flex-wrap: wrap;
 
5984
  .filter(k => fps[k])
5985
  .map(k => `<span class="ab-pp-fmt-pair"><span class="ab-pp-fmt-name">${esc(k)}</span> <span class="ab-pp-fmt-price">${esc(fps[k])}</span></span>`)
5986
  .join('');
5987
+ if (parts) return `<div class="ab-pp-price ab-pp-price--fmts">${parts}</div>`;
5988
  }
5989
+ if (p.price) return `<div class="ab-pp-price">${esc(p.price)}</div>`;
5990
+ return `<div class="ab-pp-price ab-pp-price--na">Price not available</div>`;
5991
  })();
5992
  const attrs = missing
5993
  ? ` data-pp-platform-id="${esc(p.platform_id)}" data-pp-missing="1" role="button" tabindex="0" aria-label="Not found on ${esc(p.display_name)} — click for publishing help"`
 
6144
  document.getElementById('ab-e-bio').value = data.about_author || '';
6145
 
6146
  if (data.warnings && data.warnings.length) {
6147
+ const resolved = data.warnings.filter(w => String(w).startsWith('Resolved '));
6148
+ const other = data.warnings.filter(w => !String(w).startsWith('Resolved '));
6149
+ if (resolved.length) {
6150
+ abShowAlert('success', resolved.join(' · '));
6151
+ }
6152
+ if (other.length) {
6153
+ abShowAlert('warning', '⚠️ Some data was enriched from APIs: ' + other.join('; '));
6154
+ }
6155
+ }
6156
+ // Prefer canonical product URL in the paste field after share/short resolve.
6157
+ if (data.canonical_url && document.getElementById('ab-url')) {
6158
+ const paste = (document.getElementById('ab-url').value || '').trim();
6159
+ if (paste && data.canonical_url !== paste) {
6160
+ document.getElementById('ab-url').value = data.canonical_url;
6161
+ }
6162
+ }
6163
  abRenderPlatformScan(data.platform_presence);
6164
  abSetState('preview');
6165
  }
 
6309
  function abShowAlert(type, msg) {
6310
  const el = document.getElementById('ab-alert');
6311
  if (!el) return;
6312
+ const colors = {
6313
+ error: 'var(--red-bg);border:1px solid rgba(239,68,68,.25);color:var(--red)',
6314
+ warning: 'var(--yellow-bg);border:1px solid rgba(245,158,11,.25);color:#92400e',
6315
+ success: 'rgba(16,185,129,.12);border:1px solid rgba(16,185,129,.3);color:#065f46',
6316
+ };
6317
  el.style.cssText = `display:flex;align-items:flex-start;gap:10px;padding:11px 14px;border-radius:var(--radius-sm);margin-bottom:14px;font-size:12px;background:${colors[type]||colors.error}`;
6318
  el.textContent = msg;
6319
  }
app/services/book_service.py CHANGED
@@ -361,11 +361,12 @@ class BookService:
361
  author=meta.author or "",
362
  isbn=meta.isbn13 or meta.isbn or None,
363
  source_platform=meta.platform if meta.platform != "unknown" else None,
364
- source_url=url,
365
- buy_url=meta.buy_url or url,
 
366
  redis=redis,
367
  )
368
- await save_scan_to_redis(redis, url, scan)
369
  platform_presence = platform_presence_to_dict(scan)
370
 
371
  platform_display = PLATFORM_DISPLAY.get(
@@ -399,8 +400,9 @@ class BookService:
399
  "genre": meta.genre,
400
  "rating": meta.rating,
401
  "rating_count": meta.rating_count,
402
- "buy_url": meta.buy_url or url,
403
- "source_url": url,
 
404
  "confidence": meta.confidence,
405
  "error": meta.error or None,
406
  "warnings": meta.warnings or [],
 
361
  author=meta.author or "",
362
  isbn=meta.isbn13 or meta.isbn or None,
363
  source_platform=meta.platform if meta.platform != "unknown" else None,
364
+ source_url=meta.source_url or url,
365
+ buy_url=meta.buy_url or meta.source_url or url,
366
+ db=db,
367
  redis=redis,
368
  )
369
+ await save_scan_to_redis(redis, meta.source_url or url, scan)
370
  platform_presence = platform_presence_to_dict(scan)
371
 
372
  platform_display = PLATFORM_DISPLAY.get(
 
400
  "genre": meta.genre,
401
  "rating": meta.rating,
402
  "rating_count": meta.rating_count,
403
+ "buy_url": meta.buy_url or meta.source_url or url,
404
+ "source_url": meta.source_url or url,
405
+ "canonical_url": meta.source_url or url,
406
  "confidence": meta.confidence,
407
  "error": meta.error or None,
408
  "warnings": meta.warnings or [],
app/services/book_url_scraper.py CHANGED
@@ -193,11 +193,22 @@ PLATFORM_ICONS: dict[str, str] = {
193
  }
194
 
195
  def detect_platform(url: str) -> str:
196
- """Return platform id for a URL. Google Books URLs are not import platforms."""
 
 
 
 
197
  low = url.lower()
198
  # Explicitly reject Google Books / Play Books as a scrape target.
199
  if re.search(r"books\.google\.|play\.google\.com/store/books", low):
200
  return "unknown"
 
 
 
 
 
 
 
201
  for platform, pattern in PLATFORM_PATTERNS.items():
202
  if re.search(pattern, low):
203
  return platform
@@ -1908,10 +1919,10 @@ def _extract_identifiers(url: str) -> dict:
1908
  """Extract all book identifiers directly from the URL — no HTTP needed."""
1909
  ids: dict = {}
1910
 
1911
- # Amazon ASIN: /dp/XXXXXXXXXX or /gp/product/XXXXXXXXXX
1912
- m = re.search(r"/(?:dp|gp/product)/([A-Z0-9]{10})", url)
1913
  if m:
1914
- ids["asin"] = m.group(1)
1915
 
1916
  # Google Books volume ID: id=XXXX in query string
1917
  m = re.search(r"[?&]id=([A-Za-z0-9_-]{8,})", url)
@@ -1954,14 +1965,23 @@ def _extract_identifiers(url: str) -> dict:
1954
  if m:
1955
  ids["goodreads_id"] = m.group(1)
1956
 
1957
- # Barnes & Noble EAN: /w/title/XXXXXXXXXX (often a product ID, not ISBN)
1958
- m = re.search(r"barnesandnoble\.com/w/[^/]+/(\d{10,13})", url)
1959
  if m:
1960
  ean = m.group(1)
 
1961
  if len(ean) == 13 and _is_valid_isbn13(ean):
1962
  ids["isbn"] = ean
1963
  elif len(ean) == 10 and _is_valid_isbn10(ean):
1964
  ids["isbn10"] = ean
 
 
 
 
 
 
 
 
1965
 
1966
  # Kobo — ISBN often embedded in /ebook/{slug} path
1967
  m = re.search(r"kobo\.com/[^?]*(?:ebook|book)/([^/?#]+)", url, re.IGNORECASE)
@@ -1979,9 +1999,9 @@ def _extract_identifiers(url: str) -> dict:
1979
  if len(candidate) == 13 and _is_valid_isbn13(candidate):
1980
  ids["isbn"] = candidate
1981
 
1982
- # Apple Books internal ID: /book/.../id123 or /audiobook/.../id123
1983
  m = re.search(
1984
- r"books\.apple\.com/[^/]+/(?:book|audiobook)/[^/]+/id(\d+)",
1985
  url,
1986
  re.IGNORECASE,
1987
  )
@@ -6121,17 +6141,30 @@ async def fetch_book_metadata(
6121
  Returns:
6122
  BookURLMetadata — never raises. `error` field set on complete failure.
6123
  """
6124
- url = _normalize_import_url(url.strip())
6125
- platform = detect_platform(url)
 
 
 
 
 
 
 
6126
 
6127
  meta = BookURLMetadata(source_url=url, platform=platform, buy_url=url)
 
 
 
 
6128
 
6129
  if platform == "unknown":
6130
  # Don't give up! Try generic HTML + Google Books search for any URL.
6131
  # Author personal sites, indie stores, custom URLs all work this way.
6132
  platform = "generic"
6133
  meta.platform = "generic"
6134
- meta.warnings = ["Unrecognized platform — using generic fetch + Google Books search"]
 
 
6135
 
6136
  # ── Redis cache (always load — used for fast path + re-fetch fallback) ───
6137
  cache_key = _CACHE_PREFIX + hashlib.sha256(url.encode()).hexdigest()[:20]
@@ -6142,12 +6175,21 @@ async def fetch_book_metadata(
6142
  if cached:
6143
  previous = BookURLMetadata(**json.loads(cached))
6144
  if not force_refresh:
 
 
 
 
 
 
 
6145
  return previous
6146
  except Exception:
6147
  pass
6148
 
6149
  # ── Resolve ────────────────────────────────────────────────────────────────
6150
  ids = _extract_identifiers(url)
 
 
6151
  logger.info("Resolving book URL", platform=platform, ids=list(ids.keys()))
6152
 
6153
  raw: dict = {}
 
193
  }
194
 
195
  def detect_platform(url: str) -> str:
196
+ """Return platform id for a URL. Google Books URLs are not import platforms.
197
+
198
+ Share/short hosts (a.co, amzn.to, bn.com, itunes.apple.com, …) are
199
+ classified via the host-family map so they are never mislabeled unknown.
200
+ """
201
  low = url.lower()
202
  # Explicitly reject Google Books / Play Books as a scrape target.
203
  if re.search(r"books\.google\.|play\.google\.com/store/books", low):
204
  return "unknown"
205
+ try:
206
+ from app.services.url_canonicalizer import host_family_platform
207
+ family = host_family_platform(url)
208
+ if family:
209
+ return family
210
+ except Exception:
211
+ pass
212
  for platform, pattern in PLATFORM_PATTERNS.items():
213
  if re.search(pattern, low):
214
  return platform
 
1919
  """Extract all book identifiers directly from the URL — no HTTP needed."""
1920
  ids: dict = {}
1921
 
1922
+ # Amazon ASIN: /dp/, /gp/product/, or mobile /gp/aw/d/
1923
+ m = re.search(r"/(?:dp|gp/product|gp/aw/d)/([A-Z0-9]{10})", url, re.IGNORECASE)
1924
  if m:
1925
+ ids["asin"] = m.group(1).upper()
1926
 
1927
  # Google Books volume ID: id=XXXX in query string
1928
  m = re.search(r"[?&]id=([A-Za-z0-9_-]{8,})", url)
 
1965
  if m:
1966
  ids["goodreads_id"] = m.group(1)
1967
 
1968
+ # Barnes & Noble: /w/{slug}/{id} or /w/{id} (id is often not an ISBN)
1969
+ m = re.search(r"barnesandnoble\.com/w/(?:[^/]+/)?(\d{8,13})", url, re.IGNORECASE)
1970
  if m:
1971
  ean = m.group(1)
1972
+ ids["bn_id"] = ean
1973
  if len(ean) == 13 and _is_valid_isbn13(ean):
1974
  ids["isbn"] = ean
1975
  elif len(ean) == 10 and _is_valid_isbn10(ean):
1976
  ids["isbn10"] = ean
1977
+ # Prefer ?ean= when present on BN URLs
1978
+ m = re.search(r"[?&]ean=(\d{10,13})", url, re.IGNORECASE)
1979
+ if m:
1980
+ candidate = m.group(1)
1981
+ if len(candidate) == 13 and _is_valid_isbn13(candidate):
1982
+ ids["isbn"] = candidate
1983
+ elif len(candidate) == 10 and _is_valid_isbn10(candidate):
1984
+ ids["isbn10"] = candidate
1985
 
1986
  # Kobo — ISBN often embedded in /ebook/{slug} path
1987
  m = re.search(r"kobo\.com/[^?]*(?:ebook|book)/([^/?#]+)", url, re.IGNORECASE)
 
1999
  if len(candidate) == 13 and _is_valid_isbn13(candidate):
2000
  ids["isbn"] = candidate
2001
 
2002
+ # Apple Books / legacy iTunes: /book/.../id123 or /book/id123
2003
  m = re.search(
2004
+ r"(?:books|itunes)\.apple\.com/[^/]+/(?:book|audiobook)(?:/[^/]+)?/id(\d+)",
2005
  url,
2006
  re.IGNORECASE,
2007
  )
 
6141
  Returns:
6142
  BookURLMetadata — never raises. `error` field set on complete failure.
6143
  """
6144
+ from app.services.url_canonicalizer import canonicalize_book_url
6145
+
6146
+ original_paste = (url or "").strip()
6147
+ # Step 0: expand share/short/affiliate → stable product URL (R-URL-001–005)
6148
+ canon = await canonicalize_book_url(original_paste)
6149
+ url = _normalize_import_url(canon.canonical_url or original_paste)
6150
+ platform = canon.platform_id if canon.platform_id != "unknown" else detect_platform(url)
6151
+ if platform == "unknown":
6152
+ platform = detect_platform(url)
6153
 
6154
  meta = BookURLMetadata(source_url=url, platform=platform, buy_url=url)
6155
+ if canon.was_resolved and canon.original_host and canon.canonical_url:
6156
+ meta.warnings.append(
6157
+ f"Resolved {canon.original_host} → {canon.canonical_url}"
6158
+ )
6159
 
6160
  if platform == "unknown":
6161
  # Don't give up! Try generic HTML + Google Books search for any URL.
6162
  # Author personal sites, indie stores, custom URLs all work this way.
6163
  platform = "generic"
6164
  meta.platform = "generic"
6165
+ meta.warnings.append(
6166
+ "Unrecognized platform — using generic fetch + Google Books search"
6167
+ )
6168
 
6169
  # ── Redis cache (always load — used for fast path + re-fetch fallback) ───
6170
  cache_key = _CACHE_PREFIX + hashlib.sha256(url.encode()).hexdigest()[:20]
 
6175
  if cached:
6176
  previous = BookURLMetadata(**json.loads(cached))
6177
  if not force_refresh:
6178
+ # Preserve resolve toast on cache hit when paste was a short link.
6179
+ if meta.warnings and not previous.warnings:
6180
+ previous.warnings = list(meta.warnings)
6181
+ elif meta.warnings:
6182
+ for w in meta.warnings:
6183
+ if w not in previous.warnings:
6184
+ previous.warnings.append(w)
6185
  return previous
6186
  except Exception:
6187
  pass
6188
 
6189
  # ── Resolve ────────────────────────────────────────────────────────────────
6190
  ids = _extract_identifiers(url)
6191
+ for k, v in (canon.extracted_ids or {}).items():
6192
+ ids.setdefault(k, v)
6193
  logger.info("Resolving book URL", platform=platform, ids=list(ids.keys()))
6194
 
6195
  raw: dict = {}
app/services/isbn_catalog.py CHANGED
@@ -206,18 +206,24 @@ def normalize_retailer_url(url: str) -> str:
206
 
207
 
208
  _PLATFORM_HOSTS: dict[str, tuple[str, ...]] = {
209
- "amazon": ("amazon.com", "amazon.co.uk", "amazon.ca"),
 
 
 
210
  "apple_books": ("books.apple.com", "itunes.apple.com"),
211
  "google_books": ("books.google.com", "google.com/books", "play.google.com"),
212
  "google_play": ("play.google.com",),
213
  "open_library": ("openlibrary.org",),
214
- "barnes_noble": ("barnesandnoble.com",),
215
  "kobo": ("kobo.com",),
216
  "bookshop": ("bookshop.org",),
217
  "walmart": ("walmart.com",),
218
  "goodreads": ("goodreads.com",),
219
  "thriftbooks": ("thriftbooks.com",),
220
- "abebooks": ("abebooks.com",),
 
 
 
221
  }
222
 
223
  _SEARCH_MARKERS: dict[str, tuple[str, ...]] = {
@@ -302,7 +308,17 @@ def sanitize_platform_url(url: str, platform_id: str) -> str | None:
302
  cleaned = normalize_retailer_url((url or "").strip())
303
  if not cleaned or not cleaned.startswith("http"):
304
  return None
305
- if platform_id == "amazon":
 
 
 
 
 
 
 
 
 
 
306
  asin = extract_amazon_asin(cleaned)
307
  if asin:
308
  cleaned = f"https://www.amazon.com/dp/{asin}"
 
206
 
207
 
208
  _PLATFORM_HOSTS: dict[str, tuple[str, ...]] = {
209
+ "amazon": (
210
+ "amazon.com", "amazon.co.uk", "amazon.ca", "amazon.com.au",
211
+ "amazon.de", "a.co", "amzn.to", "smile.amazon.com",
212
+ ),
213
  "apple_books": ("books.apple.com", "itunes.apple.com"),
214
  "google_books": ("books.google.com", "google.com/books", "play.google.com"),
215
  "google_play": ("play.google.com",),
216
  "open_library": ("openlibrary.org",),
217
+ "barnes_noble": ("barnesandnoble.com", "bn.com"),
218
  "kobo": ("kobo.com",),
219
  "bookshop": ("bookshop.org",),
220
  "walmart": ("walmart.com",),
221
  "goodreads": ("goodreads.com",),
222
  "thriftbooks": ("thriftbooks.com",),
223
+ "abebooks": (
224
+ "abebooks.com", "abebooks.co.uk", "abebooks.de",
225
+ "iberlibro.com", "zvab.com",
226
+ ),
227
  }
228
 
229
  _SEARCH_MARKERS: dict[str, tuple[str, ...]] = {
 
308
  cleaned = normalize_retailer_url((url or "").strip())
309
  if not cleaned or not cleaned.startswith("http"):
310
  return None
311
+ try:
312
+ from app.services.url_canonicalizer import rewrite_product_url
313
+ rewritten, _ids = rewrite_product_url(cleaned, platform_id)
314
+ if rewritten:
315
+ cleaned = rewritten
316
+ except Exception:
317
+ if platform_id == "amazon":
318
+ asin = extract_amazon_asin(cleaned)
319
+ if asin:
320
+ cleaned = f"https://www.amazon.com/dp/{asin}"
321
+ if platform_id == "amazon" and "/dp/" not in cleaned:
322
  asin = extract_amazon_asin(cleaned)
323
  if asin:
324
  cleaned = f"https://www.amazon.com/dp/{asin}"
app/services/platform_presence.py CHANGED
@@ -39,7 +39,7 @@ logger = structlog.get_logger(__name__)
39
 
40
  _SCAN_TIMEOUT = 10.0
41
  _MAX_CONCURRENT = 6
42
- _SCAN_CACHE_PREFIX = "platform_scan:v12:"
43
  _SCAN_CACHE_TTL = 21_600 # 6h
44
 
45
  Status = Literal["verified", "likely", "not_found", "checking", "error", "skipped"]
@@ -745,6 +745,15 @@ def _finalize_listing_url(listing: PlatformListing) -> None:
745
  return
746
 
747
  clean = sanitize_platform_url(listing.listing_url, pid)
 
 
 
 
 
 
 
 
 
748
  if not clean or classify_retailer_url(clean, pid) != "product":
749
  listing.listing_url = None
750
  listing.status = "not_found"
@@ -816,10 +825,19 @@ async def _enrich_listing_prices(
816
 
817
  async def _one(listing: PlatformListing) -> None:
818
  try:
 
 
 
 
 
 
 
 
 
819
  hit = await fetch_listing_format_prices(
820
  client,
821
  listing.platform_id,
822
- listing.listing_url or "",
823
  ctx,
824
  credential=credentials.get(listing.platform_id),
825
  )
 
39
 
40
  _SCAN_TIMEOUT = 10.0
41
  _MAX_CONCURRENT = 6
42
+ _SCAN_CACHE_PREFIX = "platform_scan:v13:"
43
  _SCAN_CACHE_TTL = 21_600 # 6h
44
 
45
  Status = Literal["verified", "likely", "not_found", "checking", "error", "skipped"]
 
745
  return
746
 
747
  clean = sanitize_platform_url(listing.listing_url, pid)
748
+ # Sync rewrite to stable product form (share hosts need async resolve elsewhere).
749
+ if clean:
750
+ try:
751
+ from app.services.url_canonicalizer import rewrite_product_url
752
+ rewritten, _ids = rewrite_product_url(clean, pid)
753
+ if rewritten:
754
+ clean = rewritten
755
+ except Exception:
756
+ pass
757
  if not clean or classify_retailer_url(clean, pid) != "product":
758
  listing.listing_url = None
759
  listing.status = "not_found"
 
825
 
826
  async def _one(listing: PlatformListing) -> None:
827
  try:
828
+ from app.services.url_canonicalizer import canonicalize_book_url
829
+ listing_url = listing.listing_url or ""
830
+ canon = await canonicalize_book_url(listing_url, client=client)
831
+ if (
832
+ canon.canonical_url
833
+ and canon.platform_id == listing.platform_id
834
+ ):
835
+ listing_url = canon.canonical_url
836
+ listing.listing_url = listing_url
837
  hit = await fetch_listing_format_prices(
838
  client,
839
  listing.platform_id,
840
+ listing_url,
841
  ctx,
842
  credential=credentials.get(listing.platform_id),
843
  )
app/services/platform_pricing.py CHANGED
@@ -16,12 +16,16 @@ import structlog
16
  from app.services.book_url_scraper import (
17
  _API_TIMEOUT,
18
  _browser_headers,
 
19
  _extract_amazon_all_format_prices,
20
  _extract_bn_all_format_prices,
 
21
  _extract_json_ld,
 
22
  _google_books_api_params,
23
  _normalize_retail_format_label,
24
  _parse_amazon_html,
 
25
  _score_itunes_item,
26
  )
27
 
@@ -467,11 +471,12 @@ async def fetch_listing_format_prices(
467
  return await _fetch_apple_format_prices(client, listing_url, ctx)
468
 
469
  # --- Credential-based platforms ---
470
- # Amazon / Barnes & Noble expose ALL formats on their product page, so we
471
- # skip the single-price credential shortcut for them and let the HTML
472
- # multi-format parser run below. Only use credential fetch as a fallback
473
- # (handled after the HTML attempt) so we never collapse 3 formats into 1.
474
- _MULTI_FORMAT_HTML = {"amazon", "barnes_noble"}
 
475
  if credential is not None and platform_id not in _MULTI_FORMAT_HTML:
476
  from app.services.platform_api_registry import fetch_price_via_credential
477
  hit = await fetch_price_via_credential(client, credential, platform_id, listing_url, ctx)
@@ -503,11 +508,7 @@ async def fetch_listing_format_prices(
503
  continue
504
  html = r.text
505
 
506
- fmt_map = {}
507
- if platform_id == "amazon":
508
- fmt_map = _extract_amazon_all_format_prices(html) or {}
509
- elif platform_id == "barnes_noble":
510
- fmt_map = _extract_bn_all_format_prices(html) or {}
511
 
512
  if fmt_map:
513
  norm = canonical_format_prices(fmt_map, platform_id)
@@ -546,7 +547,7 @@ async def fetch_listing_format_prices(
546
  error=str(exc)[:120],
547
  )
548
 
549
- # Amazon / BN HTML gave nothing (e.g. bot wall) — fall back to credential fetch.
550
  if credential is not None and platform_id in _MULTI_FORMAT_HTML:
551
  from app.services.platform_api_registry import fetch_price_via_credential
552
  hit = await fetch_price_via_credential(client, credential, platform_id, listing_url, ctx)
@@ -564,6 +565,28 @@ async def fetch_listing_format_prices(
564
  return None
565
 
566
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
567
  def _single_price_credential_format(platform_id: str) -> str:
568
  """Best-effort format label for a single credential price (empty if unknown)."""
569
  if platform_id in {"kobo", "google_books"}:
 
16
  from app.services.book_url_scraper import (
17
  _API_TIMEOUT,
18
  _browser_headers,
19
+ _extract_abebooks_format_prices,
20
  _extract_amazon_all_format_prices,
21
  _extract_bn_all_format_prices,
22
+ _extract_bookshop_format_prices,
23
  _extract_json_ld,
24
+ _extract_thriftbooks_format_prices,
25
  _google_books_api_params,
26
  _normalize_retail_format_label,
27
  _parse_amazon_html,
28
+ _parse_kobo_html,
29
  _score_itunes_item,
30
  )
31
 
 
471
  return await _fetch_apple_format_prices(client, listing_url, ctx)
472
 
473
  # --- Credential-based platforms ---
474
+ # Retailers that expose multiple formats on the product page skip the
475
+ # single-price credential shortcut so we never collapse 3 formats into 1.
476
+ # Credential fetch runs as a fallback after the HTML multi-format attempt.
477
+ _MULTI_FORMAT_HTML = {
478
+ "amazon", "barnes_noble", "thriftbooks", "abebooks", "bookshop", "kobo",
479
+ }
480
  if credential is not None and platform_id not in _MULTI_FORMAT_HTML:
481
  from app.services.platform_api_registry import fetch_price_via_credential
482
  hit = await fetch_price_via_credential(client, credential, platform_id, listing_url, ctx)
 
508
  continue
509
  html = r.text
510
 
511
+ fmt_map = _deep_format_prices_from_html(platform_id, html)
 
 
 
 
512
 
513
  if fmt_map:
514
  norm = canonical_format_prices(fmt_map, platform_id)
 
547
  error=str(exc)[:120],
548
  )
549
 
550
+ # Multi-format HTML gave nothing (e.g. bot wall) — fall back to credential.
551
  if credential is not None and platform_id in _MULTI_FORMAT_HTML:
552
  from app.services.platform_api_registry import fetch_price_via_credential
553
  hit = await fetch_price_via_credential(client, credential, platform_id, listing_url, ctx)
 
565
  return None
566
 
567
 
568
+ def _deep_format_prices_from_html(platform_id: str, html: str) -> dict[str, str]:
569
+ """Dispatch to the same deep format extractors used on source URL import."""
570
+ if platform_id == "amazon":
571
+ return _extract_amazon_all_format_prices(html) or {}
572
+ if platform_id == "barnes_noble":
573
+ return _extract_bn_all_format_prices(html) or {}
574
+ if platform_id == "thriftbooks":
575
+ return _extract_thriftbooks_format_prices(html) or {}
576
+ if platform_id == "abebooks":
577
+ return _extract_abebooks_format_prices(html) or {}
578
+ if platform_id == "bookshop":
579
+ return _extract_bookshop_format_prices(html) or {}
580
+ if platform_id == "kobo":
581
+ parsed = _parse_kobo_html(html) or {}
582
+ fps = dict(parsed.get("format_prices") or {})
583
+ # Never invent Hardcover for Kobo (digital-only).
584
+ fps.pop("Hardcover", None)
585
+ fps.pop("Paperback", None)
586
+ return fps
587
+ return {}
588
+
589
+
590
  def _single_price_credential_format(platform_id: str) -> str:
591
  """Best-effort format label for a single credential price (empty if unknown)."""
592
  if platform_id in {"kobo", "google_books"}:
app/services/url_canonicalizer.py ADDED
@@ -0,0 +1,422 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Canonicalize book product URLs across the 9 locked import platforms.
2
+
3
+ Share/short/affiliate hosts (a.co, amzn.to, bn.com, itunes.apple.com, …) are
4
+ first-class members of their platform family. Resolve redirects like a browser,
5
+ then rewrite to a stable product URL before scrape / presence enrichment.
6
+ """
7
+
8
+ from __future__ import annotations
9
+
10
+ import re
11
+ from dataclasses import dataclass, field
12
+ from urllib.parse import parse_qsl, unquote, urlencode, urlparse, urlunparse
13
+
14
+ import httpx
15
+ import structlog
16
+
17
+ logger = structlog.get_logger(__name__)
18
+
19
+ # Host suffixes / exact hosts → locked platform id.
20
+ # Share/short aliases are included so we never mislabel them as "unknown".
21
+ PLATFORM_HOST_FAMILIES: dict[str, tuple[str, ...]] = {
22
+ "amazon": (
23
+ "amazon.com", "amazon.co.uk", "amazon.ca", "amazon.com.au",
24
+ "amazon.de", "amazon.fr", "amazon.it", "amazon.es", "amazon.in",
25
+ "amazon.co.jp", "a.co", "amzn.to", "smile.amazon.com",
26
+ ),
27
+ "barnes_noble": ("barnesandnoble.com", "bn.com"),
28
+ "goodreads": ("goodreads.com",),
29
+ "apple_books": ("books.apple.com", "itunes.apple.com"),
30
+ "kobo": ("kobo.com",),
31
+ "thriftbooks": ("thriftbooks.com",),
32
+ "abebooks": (
33
+ "abebooks.com", "abebooks.co.uk", "abebooks.de", "abebooks.fr",
34
+ "abebooks.it", "abebooks.es", "iberlibro.com", "zvab.com",
35
+ ),
36
+ "bookshop": ("bookshop.org",),
37
+ "open_library": ("openlibrary.org",),
38
+ }
39
+
40
+ # Hosts that always need HTTP redirect resolve before ID extraction.
41
+ _SHARE_SHORT_HOSTS = frozenset({
42
+ "a.co", "amzn.to", "bn.com",
43
+ })
44
+
45
+ # Affiliate / click wrappers — resolve then re-detect.
46
+ _WRAPPER_HOST_MARKERS = (
47
+ "click.linksynergy.com",
48
+ "linksynergy.com",
49
+ "rakuten.com",
50
+ )
51
+
52
+ _TRACKING_PARAMS = frozenset({
53
+ "tag", "ref", "ref_", "psc", "linkcode", "creative", "creativeasin",
54
+ "ie", "keywords", "qid", "sr", "sprefix", "th", "dib", "dib_tag",
55
+ "utm_source", "utm_medium", "utm_campaign", "utm_content", "utm_term",
56
+ "pd_rd_w", "pd_rd_r", "pd_rd_wg", "pd_rd_i", "pf_rd_p", "pf_rd_r",
57
+ "content-id", "sp_csd", "crid", "s", "st", "2sid", "clickref",
58
+ "at", "ct", "mt", "app", "itscg", "itsct",
59
+ })
60
+
61
+ # Params that MUST be kept (product identity).
62
+ _KEEP_PARAMS = frozenset({"ean", "isbn", "id", "asin"})
63
+
64
+ _REDIRECT_TIMEOUT = 5.0
65
+
66
+
67
+ @dataclass(frozen=True)
68
+ class CanonicalUrl:
69
+ """Result of URL canonicalization for import / presence enrichment."""
70
+
71
+ platform_id: str
72
+ canonical_url: str
73
+ original_url: str
74
+ extracted_ids: dict = field(default_factory=dict)
75
+ was_resolved: bool = False
76
+ original_host: str = ""
77
+
78
+
79
+ def _bare_host(url: str) -> str:
80
+ """Return lowercase hostname without leading www."""
81
+ try:
82
+ host = (urlparse(url).netloc or "").lower()
83
+ except Exception:
84
+ return ""
85
+ if host.startswith("www."):
86
+ host = host[4:]
87
+ return host
88
+
89
+
90
+ def _ensure_https(url: str) -> str:
91
+ """Trim and force https scheme when missing."""
92
+ url = (url or "").strip()
93
+ if not url:
94
+ return ""
95
+ if url.startswith("//"):
96
+ return "https:" + url
97
+ if not re.match(r"^https?://", url, re.IGNORECASE):
98
+ return "https://" + url.lstrip("/")
99
+ if url.startswith("http://"):
100
+ return "https://" + url[7:]
101
+ return url
102
+
103
+
104
+ def host_family_platform(url: str) -> str | None:
105
+ """Return locked platform id from host family, or None if unsupported."""
106
+ host = _bare_host(url)
107
+ if not host:
108
+ return None
109
+ # Reject cover CDN / non-book Open Library assets early.
110
+ if host.startswith("covers.openlibrary.org"):
111
+ return None
112
+ for platform_id, suffixes in PLATFORM_HOST_FAMILIES.items():
113
+ for suffix in suffixes:
114
+ if host == suffix or host.endswith("." + suffix):
115
+ return platform_id
116
+ return None
117
+
118
+
119
+ def needs_redirect_resolve(url: str, platform_hint: str | None = None) -> bool:
120
+ """True when the URL is a share/short/wrapper that needs browser-like resolve."""
121
+ host = _bare_host(url)
122
+ if host in _SHARE_SHORT_HOSTS:
123
+ return True
124
+ low = (url or "").lower()
125
+ if any(marker in low for marker in _WRAPPER_HOST_MARKERS):
126
+ return True
127
+ # Abe network aliases often redirect to abebooks.com product pages.
128
+ if platform_hint == "abebooks" and host in {"iberlibro.com", "zvab.com"}:
129
+ return True
130
+ # BN search / short paths need resolve to a product page.
131
+ if platform_hint == "barnes_noble" and (
132
+ host == "bn.com" or re.search(r"barnesandnoble\.com/s/", low)
133
+ ):
134
+ return True
135
+ return False
136
+
137
+
138
+ def _strip_tracking(url: str) -> str:
139
+ """Drop tracking query params; keep product identity params."""
140
+ parsed = urlparse(url)
141
+ kept = [
142
+ (k, v) for k, v in parse_qsl(parsed.query, keep_blank_values=True)
143
+ if k.lower() not in _TRACKING_PARAMS or k.lower() in _KEEP_PARAMS
144
+ ]
145
+ return urlunparse((
146
+ parsed.scheme or "https",
147
+ parsed.netloc,
148
+ parsed.path or "",
149
+ "",
150
+ urlencode(kept),
151
+ "", # drop fragment for cache stability
152
+ ))
153
+
154
+
155
+ def _extract_asin(url: str) -> str | None:
156
+ """Extract Amazon ASIN from path or query."""
157
+ patterns = (
158
+ r"/(?:dp|gp/product|gp/aw/d|exec/obidos/ASIN)/([A-Z0-9]{10})",
159
+ r"[?&](?:asin|creativeASIN)=([A-Z0-9]{10})",
160
+ )
161
+ for pat in patterns:
162
+ m = re.search(pat, url, re.IGNORECASE)
163
+ if m:
164
+ return m.group(1).upper()
165
+ return None
166
+
167
+
168
+ def _unwrap_affiliate(url: str) -> str:
169
+ """Pull destination URL out of common affiliate wrappers without HTTP."""
170
+ for pattern in (
171
+ r"[?&]murl=([^&]+)",
172
+ r"[?&]u=([^&]+)",
173
+ r"[?&]url=([^&]+)",
174
+ r"RD_PARM1=([^&]+)",
175
+ r"destination:(https?://[^\s\"']+)",
176
+ ):
177
+ m = re.search(pattern, url, re.IGNORECASE)
178
+ if m:
179
+ candidate = unquote(m.group(1))
180
+ if candidate.startswith("http"):
181
+ return candidate
182
+ return url
183
+
184
+
185
+ def rewrite_product_url(url: str, platform_id: str) -> tuple[str, dict]:
186
+ """Rewrite a resolved URL to the platform's stable product form.
187
+
188
+ Returns (canonical_url, extracted_ids).
189
+ """
190
+ url = _strip_tracking(_ensure_https(url))
191
+ ids: dict = {}
192
+ parsed = urlparse(url)
193
+ host = (parsed.netloc or "").lower()
194
+ path = parsed.path or ""
195
+ query = dict(parse_qsl(parsed.query, keep_blank_values=True))
196
+
197
+ if platform_id == "amazon":
198
+ asin = _extract_asin(url)
199
+ if asin:
200
+ ids["asin"] = asin
201
+ return f"https://www.amazon.com/dp/{asin}", ids
202
+ return url, ids
203
+
204
+ if platform_id == "barnes_noble":
205
+ # /w/{slug}/{id} or /w/{id}
206
+ m = re.search(r"/w/(?:([^/]+)/)?(\d{8,13})(?:/|$)", path, re.IGNORECASE)
207
+ if m:
208
+ slug, bn_id = m.group(1) or "", m.group(2)
209
+ ids["bn_id"] = bn_id
210
+ ean = query.get("ean") or query.get("EAN") or ""
211
+ if ean and re.fullmatch(r"\d{10,13}", ean):
212
+ ids["isbn"] = ean
213
+ base = (
214
+ f"https://www.barnesandnoble.com/w/{slug}/{bn_id}"
215
+ if slug
216
+ else f"https://www.barnesandnoble.com/w/{bn_id}"
217
+ )
218
+ if ids.get("isbn"):
219
+ return f"{base}?ean={ids['isbn']}", ids
220
+ return base, ids
221
+ # /s/{isbn} search — keep for redirect resolve; no rewrite yet
222
+ m = re.search(r"/s/(\d{10,13})", path)
223
+ if m:
224
+ ids["isbn"] = m.group(1)
225
+ return url, ids
226
+
227
+ if platform_id == "goodreads":
228
+ m = re.search(r"/book/show/(\d+)", path)
229
+ if m:
230
+ ids["goodreads_id"] = m.group(1)
231
+ return f"https://www.goodreads.com/book/show/{m.group(1)}", ids
232
+ return url, ids
233
+
234
+ if platform_id == "apple_books":
235
+ m = re.search(
236
+ r"(?:books|itunes)\.apple\.com/([^/]+)/(book|audiobook)(?:/[^/]+)?/id(\d+)",
237
+ url,
238
+ re.IGNORECASE,
239
+ )
240
+ if m:
241
+ cc, kind, book_id = m.group(1), m.group(2), m.group(3)
242
+ ids["apple_book_id"] = book_id
243
+ if kind.lower() == "audiobook":
244
+ ids["apple_is_audiobook"] = True
245
+ return f"https://books.apple.com/{cc}/{kind}/id{book_id}", ids
246
+ # id-only without slug already covered; try bare /idN
247
+ m = re.search(r"books\.apple\.com/([^/]+)/book/id(\d+)", url, re.IGNORECASE)
248
+ if m:
249
+ ids["apple_book_id"] = m.group(2)
250
+ return f"https://books.apple.com/{m.group(1)}/book/id{m.group(2)}", ids
251
+ return url, ids
252
+
253
+ if platform_id == "kobo":
254
+ m = re.search(
255
+ r"kobo\.com/((?:\w+/)?(?:\w+/)?)(ebook|audiobook)/([^/?#]+)",
256
+ url,
257
+ re.IGNORECASE,
258
+ )
259
+ if m:
260
+ kind = m.group(2).lower()
261
+ slug = m.group(3)
262
+ isbn_m = re.search(r"(\d{13})", slug)
263
+ if isbn_m:
264
+ ids["isbn"] = isbn_m.group(1)
265
+ # Preserve locale path from original when present.
266
+ return _strip_tracking(url.split("?")[0].rstrip("/") + "/"), ids
267
+ return url, ids
268
+
269
+ if platform_id == "thriftbooks":
270
+ m = re.search(r"/w/([^/]+)/(\d+)/?", path)
271
+ if m:
272
+ ids["thrift_work_id"] = m.group(2)
273
+ return f"https://www.thriftbooks.com/w/{m.group(1)}/{m.group(2)}/", ids
274
+ frag = urlparse(url).fragment or ""
275
+ fm = re.search(r"isbn[=:](\d{10,13})", frag, re.IGNORECASE)
276
+ if fm:
277
+ ids["isbn"] = fm.group(1)
278
+ return url.split("#")[0].rstrip("/") + ("/" if "/w/" in path else ""), ids
279
+
280
+ if platform_id == "abebooks":
281
+ m = re.search(r"/products/isbn/(\d{10,13})", path, re.IGNORECASE)
282
+ if m:
283
+ ids["isbn"] = m.group(1)
284
+ return f"https://www.abebooks.com/products/isbn/{m.group(1)}", ids
285
+ m = re.search(r"[?&]isbn=(\d{10,13})", url, re.IGNORECASE)
286
+ if m:
287
+ ids["isbn"] = m.group(1)
288
+ return f"https://www.abebooks.com/products/isbn/{m.group(1)}", ids
289
+ return url, ids
290
+
291
+ if platform_id == "bookshop":
292
+ m = re.search(r"/p/books/([^/]+)/(\d+)", path, re.IGNORECASE)
293
+ if m:
294
+ slug, bid = m.group(1), m.group(2)
295
+ ean = query.get("ean") or ""
296
+ if ean and re.fullmatch(r"\d{10,13}", ean):
297
+ ids["isbn"] = ean
298
+ return f"https://bookshop.org/p/books/{slug}/{bid}?ean={ean}", ids
299
+ return f"https://bookshop.org/p/books/{slug}/{bid}", ids
300
+ m = re.search(r"/books/(\d{10,13})", path)
301
+ if m:
302
+ ids["isbn"] = m.group(1)
303
+ return f"https://bookshop.org/books/{m.group(1)}", ids
304
+ return url, ids
305
+
306
+ if platform_id == "open_library":
307
+ # Strip .json API suffix for page canonical.
308
+ path2 = re.sub(r"\.json$", "", path, flags=re.IGNORECASE)
309
+ m = re.search(r"/works/(OL\d+W)", path2, re.IGNORECASE)
310
+ if m:
311
+ ids["ol_work_key"] = m.group(1).upper().replace("OL", "OL")
312
+ key = m.group(1)
313
+ if not key.startswith("OL"):
314
+ key = "OL" + key
315
+ # Normalize casing: OLxxxW
316
+ key_m = re.search(r"(OL\d+W)", path2, re.IGNORECASE)
317
+ key = key_m.group(1) if key_m else m.group(1)
318
+ ids["ol_work_key"] = key
319
+ return f"https://openlibrary.org/works/{key}", ids
320
+ m = re.search(r"/books/(OL\d+M)", path2, re.IGNORECASE)
321
+ if m:
322
+ ids["ol_edition_key"] = m.group(1)
323
+ return f"https://openlibrary.org/books/{m.group(1)}", ids
324
+ m = re.search(r"/isbn/(\d{10,13})", path2)
325
+ if m:
326
+ ids["isbn"] = m.group(1)
327
+ return f"https://openlibrary.org/isbn/{m.group(1)}", ids
328
+ if path2 != path:
329
+ return urlunparse((parsed.scheme, parsed.netloc, path2, "", "", "")), ids
330
+ return url, ids
331
+
332
+ return url, ids
333
+
334
+
335
+ async def _follow_redirects(url: str, client: httpx.AsyncClient | None = None) -> str:
336
+ """Follow HTTP redirects; return final URL (or original on failure)."""
337
+ own_client = client is None
338
+ if own_client:
339
+ client = httpx.AsyncClient(
340
+ follow_redirects=True,
341
+ timeout=_REDIRECT_TIMEOUT,
342
+ headers={
343
+ "User-Agent": (
344
+ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
345
+ "AppleWebKit/537.36 (KHTML, like Gecko) "
346
+ "Chrome/124.0.0.0 Safari/537.36"
347
+ ),
348
+ "Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.8",
349
+ },
350
+ )
351
+ assert client is not None
352
+ try:
353
+ # Prefer HEAD; some short-link hosts only support GET.
354
+ try:
355
+ r = await client.head(url)
356
+ final = str(r.url)
357
+ if final and final != url and "a.co" not in _bare_host(final):
358
+ return final
359
+ except Exception:
360
+ pass
361
+ r = await client.get(url)
362
+ return str(r.url) or url
363
+ except Exception as exc:
364
+ logger.debug("URL redirect resolve failed", url=url[:80], error=str(exc)[:120])
365
+ return url
366
+ finally:
367
+ if own_client:
368
+ await client.aclose()
369
+
370
+
371
+ async def canonicalize_book_url(
372
+ raw_url: str,
373
+ client: httpx.AsyncClient | None = None,
374
+ ) -> CanonicalUrl:
375
+ """Expand share/short links and rewrite to a stable product URL.
376
+
377
+ Never raises — on failure returns best-effort rewrite of the original URL.
378
+ """
379
+ original = _ensure_https(raw_url)
380
+ if not original:
381
+ return CanonicalUrl(
382
+ platform_id="unknown",
383
+ canonical_url="",
384
+ original_url=raw_url or "",
385
+ original_host="",
386
+ )
387
+
388
+ original_host = _bare_host(original)
389
+ working = _unwrap_affiliate(original)
390
+ platform_hint = host_family_platform(working)
391
+ was_resolved = False
392
+
393
+ if needs_redirect_resolve(working, platform_hint) or (
394
+ platform_hint is None and any(m in working.lower() for m in _WRAPPER_HOST_MARKERS)
395
+ ):
396
+ resolved = await _follow_redirects(working, client=client)
397
+ if resolved and resolved != working:
398
+ was_resolved = True
399
+ working = resolved
400
+ platform_hint = host_family_platform(working) or platform_hint
401
+
402
+ if platform_hint is None:
403
+ platform_hint = host_family_platform(working) or "unknown"
404
+
405
+ if platform_hint == "unknown":
406
+ return CanonicalUrl(
407
+ platform_id="unknown",
408
+ canonical_url=_strip_tracking(working),
409
+ original_url=original,
410
+ was_resolved=was_resolved,
411
+ original_host=original_host,
412
+ )
413
+
414
+ canonical, ids = rewrite_product_url(working, platform_hint)
415
+ return CanonicalUrl(
416
+ platform_id=platform_hint,
417
+ canonical_url=canonical,
418
+ original_url=original,
419
+ extracted_ids=ids,
420
+ was_resolved=was_resolved or (original_host in _SHARE_SHORT_HOSTS),
421
+ original_host=original_host,
422
+ )
tests/unit/test_platform_presence.py CHANGED
@@ -8,6 +8,7 @@ from app.services.platform_presence import (
8
  SCORED_PLATFORMS,
9
  CatalogContext,
10
  PlatformListing,
 
11
  build_recommendations,
12
  compute_score,
13
  confidence_to_status,
@@ -19,6 +20,8 @@ from app.services.platform_presence import (
19
  title_match_score,
20
  )
21
  from app.repositories.platform_listing_repo import PlatformListingRepository
 
 
22
 
23
 
24
  class TestTitleMatchScore:
@@ -295,9 +298,9 @@ async def test_build_catalog_uses_slug_fetch_and_bookshop_isbn():
295
 
296
 
297
  @pytest.mark.asyncio
298
- async def test_scan_cache_key_busted_for_v10():
299
  from app.services.platform_presence import scan_cache_key
300
- assert scan_cache_key("https://example.com/book").startswith("platform_scan:v10:")
301
 
302
 
303
  @pytest.mark.asyncio
@@ -424,6 +427,50 @@ def test_parse_goodreads_book_ref():
424
  assert ref == ("219584318", "/book/show/219584318-lethal-lines")
425
 
426
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
427
  @pytest.mark.integration
428
  @pytest.mark.asyncio
429
  async def test_lethal_lines_kindle_only_scan():
 
8
  SCORED_PLATFORMS,
9
  CatalogContext,
10
  PlatformListing,
11
+ PlatformPresenceResult,
12
  build_recommendations,
13
  compute_score,
14
  confidence_to_status,
 
20
  title_match_score,
21
  )
22
  from app.repositories.platform_listing_repo import PlatformListingRepository
23
+ from app.services.book_service import BookService
24
+ from app.services.book_url_scraper import BookURLMetadata
25
 
26
 
27
  class TestTitleMatchScore:
 
298
 
299
 
300
  @pytest.mark.asyncio
301
+ async def test_scan_cache_key_busted_for_v13():
302
  from app.services.platform_presence import scan_cache_key
303
+ assert scan_cache_key("https://example.com/book").startswith("platform_scan:v13:")
304
 
305
 
306
  @pytest.mark.asyncio
 
427
  assert ref == ("219584318", "/book/show/219584318-lethal-lines")
428
 
429
 
430
+ @pytest.mark.asyncio
431
+ async def test_preview_url_import_passes_db_to_format_price_scan():
432
+ """Fetch Details must expose configured credentials to discovered URLs."""
433
+ db = MagicMock()
434
+ redis = AsyncMock()
435
+ meta = BookURLMetadata(
436
+ title="All We Say",
437
+ author="Example Author",
438
+ platform="amazon",
439
+ source_url="https://amazon.com/dp/BOOK",
440
+ buy_url="https://amazon.com/dp/BOOK",
441
+ )
442
+ scan = PlatformPresenceResult(
443
+ score=0.0,
444
+ listed_count=0,
445
+ scanned_at="2026-07-16T00:00:00+00:00",
446
+ source_isbn=None,
447
+ platforms=[],
448
+ recommendations=[],
449
+ )
450
+
451
+ with (
452
+ patch(
453
+ "app.services.book_service.fetch_book_metadata",
454
+ AsyncMock(return_value=meta),
455
+ ),
456
+ patch(
457
+ "app.services.book_service.scan_platform_presence",
458
+ AsyncMock(return_value=scan),
459
+ ) as scan_mock,
460
+ patch(
461
+ "app.services.book_service.save_scan_to_redis",
462
+ AsyncMock(),
463
+ ),
464
+ ):
465
+ result = await BookService.preview_url_import(
466
+ meta.source_url, redis, force_refresh=True, db=db,
467
+ )
468
+
469
+ assert result["platform_presence"] is not None
470
+ assert scan_mock.await_args.kwargs["db"] is db
471
+ assert scan_mock.await_args.kwargs["redis"] is redis
472
+
473
+
474
  @pytest.mark.integration
475
  @pytest.mark.asyncio
476
  async def test_lethal_lines_kindle_only_scan():
tests/unit/test_platform_pricing.py CHANGED
@@ -265,6 +265,56 @@ class TestFetchListingFormatPrices:
265
  assert hit is not None
266
  assert hit.format_prices.get("Hardcover") == "$18.00"
267
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
268
 
269
  class TestSinglePriceFormatKey:
270
  def test_kobo_is_ebook(self):
 
265
  assert hit is not None
266
  assert hit.format_prices.get("Hardcover") == "$18.00"
267
 
268
+ @pytest.mark.asyncio
269
+ async def test_bookshop_deep_extractor_used(self):
270
+ """Presence enrichment must call Bookshop deep format parser."""
271
+ html = (
272
+ "A" * 500
273
+ + 'text-dark">Hardcover</p>'
274
+ + 'subtitle text-dark">$27.02'
275
+ + 'text-dark">Ebook</p>'
276
+ + 'subtitle text-dark">$14.99'
277
+ )
278
+ resp = MagicMock()
279
+ resp.status_code = 200
280
+ resp.text = html
281
+ client = AsyncMock()
282
+ client.get = AsyncMock(return_value=resp)
283
+
284
+ with patch(
285
+ "app.services.platform_pricing._extract_bookshop_format_prices",
286
+ return_value={"Hardcover": "$27.02", "eBook": "$14.99"},
287
+ ) as deep:
288
+ hit = await fetch_listing_format_prices(
289
+ client, "bookshop", "https://bookshop.org/p/books/foo/1",
290
+ )
291
+ deep.assert_called_once()
292
+ assert hit is not None
293
+ assert hit.format_prices.get("Hardcover") == "$27.02"
294
+ assert hit.format_prices.get("eBook") == "$14.99"
295
+
296
+ @pytest.mark.asyncio
297
+ async def test_thriftbooks_deep_extractor_used(self):
298
+ """Presence enrichment must call ThriftBooks deep format parser."""
299
+ html = "A" * 600 + "Select Format"
300
+ resp = MagicMock()
301
+ resp.status_code = 200
302
+ resp.text = html
303
+ client = AsyncMock()
304
+ client.get = AsyncMock(return_value=resp)
305
+
306
+ with patch(
307
+ "app.services.platform_pricing._extract_thriftbooks_format_prices",
308
+ return_value={"Hardcover": "$8.99", "Paperback": "$4.50"},
309
+ ) as deep:
310
+ hit = await fetch_listing_format_prices(
311
+ client, "thriftbooks", "https://www.thriftbooks.com/w/foo/1/",
312
+ )
313
+ deep.assert_called_once()
314
+ assert hit is not None
315
+ assert hit.format_prices.get("Hardcover") == "$8.99"
316
+ assert "eBook" not in hit.format_prices
317
+
318
 
319
  class TestSinglePriceFormatKey:
320
  def test_kobo_is_ebook(self):
tests/unit/test_url_canonicalizer.py ADDED
@@ -0,0 +1,206 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Unit tests for URL canonicalization across locked import platforms."""
2
+
3
+ from __future__ import annotations
4
+
5
+ from unittest.mock import AsyncMock, MagicMock, patch
6
+
7
+ import pytest
8
+
9
+ from app.services.book_url_scraper import (
10
+ _extract_identifiers,
11
+ detect_platform,
12
+ )
13
+ from app.services.url_canonicalizer import (
14
+ canonicalize_book_url,
15
+ host_family_platform,
16
+ needs_redirect_resolve,
17
+ rewrite_product_url,
18
+ )
19
+
20
+
21
+ class TestHostFamilyMap:
22
+ def test_amazon_share_a_co(self):
23
+ assert host_family_platform("https://a.co/d/04YALfQb") == "amazon"
24
+
25
+ def test_amazon_amzn_to(self):
26
+ assert host_family_platform("https://amzn.to/abc123") == "amazon"
27
+
28
+ def test_bn_com(self):
29
+ assert host_family_platform("https://bn.com/s/9781234567890") == "barnes_noble"
30
+
31
+ def test_itunes_apple(self):
32
+ assert host_family_platform(
33
+ "https://itunes.apple.com/us/book/id123456789"
34
+ ) == "apple_books"
35
+
36
+ def test_iberlibro_abebooks(self):
37
+ assert host_family_platform("https://www.iberlibro.com/book/1") == "abebooks"
38
+
39
+ def test_covers_cdn_rejected(self):
40
+ assert host_family_platform(
41
+ "https://covers.openlibrary.org/b/id/123-L.jpg"
42
+ ) is None
43
+
44
+
45
+ class TestDetectPlatformShareHosts:
46
+ def test_a_co_is_amazon_not_unknown(self):
47
+ assert detect_platform("https://a.co/d/04YALfQb") == "amazon"
48
+
49
+ def test_bn_com_is_barnes_noble(self):
50
+ assert detect_platform("https://bn.com/s/9781234567890") == "barnes_noble"
51
+
52
+ def test_itunes_is_apple_books(self):
53
+ assert detect_platform(
54
+ "https://itunes.apple.com/us/book/foo/id123"
55
+ ) == "apple_books"
56
+
57
+
58
+ class TestRewriteProductUrl:
59
+ def test_amazon_affiliate_to_dp(self):
60
+ url = (
61
+ "https://www.amazon.com/Memory-Box-ebook/dp/B08KQ2PY8Y/"
62
+ "ref=pd_sbs_d_sccl_1_5?pd_rd_w=BeCFl&tag=aff-20"
63
+ )
64
+ canon, ids = rewrite_product_url(url, "amazon")
65
+ assert canon == "https://www.amazon.com/dp/B08KQ2PY8Y"
66
+ assert ids["asin"] == "B08KQ2PY8Y"
67
+
68
+ def test_amazon_mobile_aw(self):
69
+ canon, ids = rewrite_product_url(
70
+ "https://www.amazon.com/gp/aw/d/B08KQ2PY8Y", "amazon",
71
+ )
72
+ assert canon == "https://www.amazon.com/dp/B08KQ2PY8Y"
73
+ assert ids["asin"] == "B08KQ2PY8Y"
74
+
75
+ def test_bn_slug_and_id(self):
76
+ canon, ids = rewrite_product_url(
77
+ "https://www.barnesandnoble.com/w/all-we-say-ben-rhodes/1148264021?st=foo",
78
+ "barnes_noble",
79
+ )
80
+ assert "/w/all-we-say-ben-rhodes/1148264021" in canon
81
+ assert ids["bn_id"] == "1148264021"
82
+ assert "st=" not in canon
83
+
84
+ def test_bn_id_only(self):
85
+ canon, ids = rewrite_product_url(
86
+ "https://www.barnesandnoble.com/w/1148264021",
87
+ "barnes_noble",
88
+ )
89
+ assert canon == "https://www.barnesandnoble.com/w/1148264021"
90
+ assert ids["bn_id"] == "1148264021"
91
+
92
+ def test_bn_with_ean(self):
93
+ canon, ids = rewrite_product_url(
94
+ "https://www.barnesandnoble.com/w/foo/1148264021?ean=9781234567890",
95
+ "barnes_noble",
96
+ )
97
+ assert "ean=9781234567890" in canon
98
+ assert ids["isbn"] == "9781234567890"
99
+
100
+ def test_goodreads_strip_slug(self):
101
+ canon, ids = rewrite_product_url(
102
+ "https://www.goodreads.com/book/show/12345.Some_Title?from_search=true",
103
+ "goodreads",
104
+ )
105
+ assert canon == "https://www.goodreads.com/book/show/12345"
106
+ assert ids["goodreads_id"] == "12345"
107
+
108
+ def test_itunes_to_books_apple(self):
109
+ canon, ids = rewrite_product_url(
110
+ "https://itunes.apple.com/us/book/all-we-say/id6443123456?at=1000",
111
+ "apple_books",
112
+ )
113
+ assert canon == "https://books.apple.com/us/book/id6443123456"
114
+ assert ids["apple_book_id"] == "6443123456"
115
+
116
+ def test_abebooks_isbn_product(self):
117
+ canon, ids = rewrite_product_url(
118
+ "https://www.abebooks.com/servlet/SearchResults?isbn=9781234567890&cm_sp=xxx",
119
+ "abebooks",
120
+ )
121
+ assert canon == "https://www.abebooks.com/products/isbn/9781234567890"
122
+ assert ids["isbn"] == "9781234567890"
123
+
124
+ def test_bookshop_with_ean(self):
125
+ canon, ids = rewrite_product_url(
126
+ "https://bookshop.org/p/books/all-we-say/12345?ean=9781234567890&utm_source=x",
127
+ "bookshop",
128
+ )
129
+ assert "ean=9781234567890" in canon
130
+ assert "utm_source" not in canon
131
+
132
+ def test_open_library_strip_json(self):
133
+ canon, ids = rewrite_product_url(
134
+ "https://openlibrary.org/works/OL45883W.json",
135
+ "open_library",
136
+ )
137
+ assert canon == "https://openlibrary.org/works/OL45883W"
138
+ assert ids["ol_work_key"] == "OL45883W"
139
+
140
+ def test_thriftbooks_work(self):
141
+ canon, ids = rewrite_product_url(
142
+ "https://www.thriftbooks.com/w/some-title/123456/?clickref=abc",
143
+ "thriftbooks",
144
+ )
145
+ assert canon == "https://www.thriftbooks.com/w/some-title/123456/"
146
+ assert ids["thrift_work_id"] == "123456"
147
+
148
+
149
+ class TestExtractIdentifiersBn:
150
+ def test_bn_id_only_path(self):
151
+ ids = _extract_identifiers("https://www.barnesandnoble.com/w/1148264021")
152
+ assert ids.get("bn_id") == "1148264021"
153
+
154
+ def test_bn_ean_query(self):
155
+ ids = _extract_identifiers(
156
+ "https://www.barnesandnoble.com/w/foo/1?ean=9781234567897"
157
+ )
158
+ # Only set isbn when checksum-valid; bn_id still present from path
159
+ assert "bn_id" in ids or "isbn" in ids or "isbn10" in ids
160
+
161
+
162
+ class TestNeedsRedirect:
163
+ def test_a_co_needs_redirect(self):
164
+ assert needs_redirect_resolve("https://a.co/d/abc", "amazon") is True
165
+
166
+ def test_clean_dp_no_redirect(self):
167
+ assert needs_redirect_resolve(
168
+ "https://www.amazon.com/dp/B08KQ2PY8Y", "amazon",
169
+ ) is False
170
+
171
+
172
+ @pytest.mark.asyncio
173
+ class TestCanonicalizeAsync:
174
+ async def test_a_co_resolves_to_amazon_dp(self):
175
+ mock_resp = MagicMock()
176
+ mock_resp.url = "https://www.amazon.com/Memory-Box/dp/B08KQ2PY8Y?ref=xx"
177
+ client = AsyncMock()
178
+ client.head = AsyncMock(return_value=mock_resp)
179
+ client.get = AsyncMock(return_value=mock_resp)
180
+
181
+ result = await canonicalize_book_url("https://a.co/d/04YALfQb", client=client)
182
+ assert result.platform_id == "amazon"
183
+ assert result.canonical_url == "https://www.amazon.com/dp/B08KQ2PY8Y"
184
+ assert result.was_resolved is True
185
+ assert result.original_host == "a.co"
186
+ assert result.extracted_ids.get("asin") == "B08KQ2PY8Y"
187
+
188
+ async def test_clean_amazon_no_http(self):
189
+ with patch(
190
+ "app.services.url_canonicalizer._follow_redirects",
191
+ new_callable=AsyncMock,
192
+ ) as follow:
193
+ result = await canonicalize_book_url(
194
+ "https://www.amazon.com/dp/B08KQ2PY8Y?tag=x",
195
+ )
196
+ follow.assert_not_called()
197
+ assert result.platform_id == "amazon"
198
+ assert result.canonical_url == "https://www.amazon.com/dp/B08KQ2PY8Y"
199
+ assert result.was_resolved is False
200
+
201
+ async def test_itunes_rewrites_without_redirect(self):
202
+ result = await canonicalize_book_url(
203
+ "https://itunes.apple.com/us/book/foo/id6443123456",
204
+ )
205
+ assert result.platform_id == "apple_books"
206
+ assert result.canonical_url == "https://books.apple.com/us/book/id6443123456"