Spaces:
Running
Running
AuthorBot Cursor commited on
Commit Β·
506f7f7
1
Parent(s): b6b3216
fix: remove Google Books from scored platforms and all author-facing UI
Browse files- .agent/BDD_SPECS.md +3 -3
- .agent/BOOK_URL_IMPORT_RULES.md +4 -3
- .agent/PLATFORM_PRICING_RULES.md +4 -3
- .agent/PRICE_INTELLIGENCE_RULES.md +2 -1
- app/admin/templates/admin.html +1 -1
- app/schemas/admin.py +2 -2
- app/services/catalog_hub.py +1 -11
- app/services/isbn_catalog.py +0 -24
- app/services/platform_presence.py +11 -34
- app/services/platform_pricing.py +3 -6
- static/admin.html +1 -1
- static/import-book.html +2 -3
- tests/bdd/features/platform_presence.feature +4 -4
- tests/bdd/steps/platform_steps.py +5 -0
- tests/unit/test_catalog_hub.py +6 -5
- tests/unit/test_platform_presence.py +5 -4
.agent/BDD_SPECS.md
CHANGED
|
@@ -408,10 +408,10 @@ Feature: Platform Presence Scan
|
|
| 408 |
And "open_library" should be verified with a product URL
|
| 409 |
And at least 5 platforms should be listed
|
| 410 |
|
| 411 |
-
Scenario: Scored platforms
|
| 412 |
When I load the scored platform list
|
| 413 |
-
Then there should be exactly
|
| 414 |
-
And the platform list should include "google_books"
|
| 415 |
And the platform list should include "open_library"
|
| 416 |
|
| 417 |
|
|
|
|
| 408 |
And "open_library" should be verified with a product URL
|
| 409 |
And at least 5 platforms should be listed
|
| 410 |
|
| 411 |
+
Scenario: Scored platforms exclude Google Books (9 channels)
|
| 412 |
When I load the scored platform list
|
| 413 |
+
Then there should be exactly 9 platforms
|
| 414 |
+
And the platform list should not include "google_books"
|
| 415 |
And the platform list should include "open_library"
|
| 416 |
|
| 417 |
|
.agent/BOOK_URL_IMPORT_RULES.md
CHANGED
|
@@ -25,9 +25,10 @@ deliberate review β not drive-by tweaks.
|
|
| 25 |
| Bookshop.org | HC / eBook / Audiobook **if listed** | curl β ScrapingBee; iTunes for pre-orders |
|
| 26 |
| Open Library | **No prices** (discovery) | Edition β parent work; HTTPS covers |
|
| 27 |
|
| 28 |
-
**Not a URL-import platform:** Google Books / Google Play Books.
|
| 29 |
-
The **Google Books API** remains an internal enricher (ISBN, volume id, title search) for
|
| 30 |
-
locked platforms. Do not re-add `google_books` to `AB_PLATFORMS`
|
|
|
|
| 31 |
|
| 32 |
Code mirror: `LOCKED_URL_IMPORT_PLATFORMS` in `app/services/book_url_scraper.py` and
|
| 33 |
`AB_PLATFORMS` in `app/admin/templates/admin.html` must stay in sync.
|
|
|
|
| 25 |
| Bookshop.org | HC / eBook / Audiobook **if listed** | curl β ScrapingBee; iTunes for pre-orders |
|
| 26 |
| Open Library | **No prices** (discovery) | Edition β parent work; HTTPS covers |
|
| 27 |
|
| 28 |
+
**Not a URL-import platform and not a scored presence channel:** Google Books / Google Play Books.
|
| 29 |
+
The **Google Books API** remains an internal enricher only (ISBN, volume id, title search) for
|
| 30 |
+
locked platforms. Do not re-add `google_books` to `AB_PLATFORMS`, `detect_platform()`, or
|
| 31 |
+
`SCORED_PLATFORMS`. Never show Google Books in Platform Presence, publish-help checklists, or bot Buy CTAs.
|
| 32 |
|
| 33 |
Code mirror: `LOCKED_URL_IMPORT_PLATFORMS` in `app/services/book_url_scraper.py` and
|
| 34 |
`AB_PLATFORMS` in `app/admin/templates/admin.html` must stay in sync.
|
.agent/PLATFORM_PRICING_RULES.md
CHANGED
|
@@ -78,14 +78,15 @@ Amazon HTML: US-locale headers + cookie (`i18n-prefs=USD`) to avoid PKR/wrong cu
|
|
| 78 |
| platform_id | Typical source | Notes |
|
| 79 |
|-------------|----------------|-------|
|
| 80 |
| amazon | credential β HTML | US headers critical |
|
| 81 |
-
| google_books | credential β API β ISBN fallback | |
|
| 82 |
| apple_books | iTunes API | `formattedPrice` |
|
| 83 |
| barnes_noble | credential β HTML JSON | |
|
| 84 |
| abebooks | credential β HTML itemprop | |
|
| 85 |
-
| kobo, thriftbooks, bookshop | credential
|
| 86 |
| goodreads | rollup | No direct price on page |
|
| 87 |
| open_library | static "Free" | Catalog semantics |
|
| 88 |
|
|
|
|
|
|
|
| 89 |
## 8. Tests
|
| 90 |
|
| 91 |
- [x] Unit β `tests/unit/test_platform_pricing.py` (format, HTML extract, API mocks)
|
|
@@ -109,7 +110,7 @@ Amazon HTML: US-locale headers + cookie (`i18n-prefs=USD`) to avoid PKR/wrong cu
|
|
| 109 |
- **R-187:** Platform scan cache key version bumped when price/credential logic changes materially
|
| 110 |
- **R-PP-010:** Scan enrichment MUST write `format_prices` when parsers find β₯1 format; MUST NOT wipe a richer existing map with a thinner scan result. Single-price fallback MUST NOT invent a binding label β Kobo may use eBook; ThriftBooks / AbeBooks / Bookshop use JSON-LD `bookFormat` when present, otherwise leave `format_prices` empty and keep `list_price` only.
|
| 111 |
- **R-PP-011:** `fetch_listing_format_prices` fetches HTML exactly once and passes the raw HTML to format-price parsers β no double HTTP round-trips per listing
|
| 112 |
-
- **R-PP-012:** Discovery platforms (goodreads, open_library
|
| 113 |
- **R-PP-013:** `fetch_listing_price` remains a backward-compatible thin wrapper over `fetch_listing_format_prices` returning only the primary formatted string
|
| 114 |
- **R-PP-014:** Apple Books enrichment MUST query iTunes `entity=ebook` and `entity=audiobook` so both digital formats can populate `format_prices` when available
|
| 115 |
- **R-PP-015:** Presence enrichment MUST call the same deep format parsers as source import β Amazon, BN, ThriftBooks, AbeBooks, Bookshop, Kobo β after one HTML fetch. Never invent missing format keys.
|
|
|
|
| 78 |
| platform_id | Typical source | Notes |
|
| 79 |
|-------------|----------------|-------|
|
| 80 |
| amazon | credential β HTML | US headers critical |
|
|
|
|
| 81 |
| apple_books | iTunes API | `formattedPrice` |
|
| 82 |
| barnes_noble | credential β HTML JSON | |
|
| 83 |
| abebooks | credential β HTML itemprop | |
|
| 84 |
+
| kobo, thriftbooks, bookshop | credential / HTML | Often 403 without ScrapingBee |
|
| 85 |
| goodreads | rollup | No direct price on page |
|
| 86 |
| open_library | static "Free" | Catalog semantics |
|
| 87 |
|
| 88 |
+
Google Books is **not** a scored or priced channel (API enricher only).
|
| 89 |
+
|
| 90 |
## 8. Tests
|
| 91 |
|
| 92 |
- [x] Unit β `tests/unit/test_platform_pricing.py` (format, HTML extract, API mocks)
|
|
|
|
| 110 |
- **R-187:** Platform scan cache key version bumped when price/credential logic changes materially
|
| 111 |
- **R-PP-010:** Scan enrichment MUST write `format_prices` when parsers find β₯1 format; MUST NOT wipe a richer existing map with a thinner scan result. Single-price fallback MUST NOT invent a binding label β Kobo may use eBook; ThriftBooks / AbeBooks / Bookshop use JSON-LD `bookFormat` when present, otherwise leave `format_prices` empty and keep `list_price` only.
|
| 112 |
- **R-PP-011:** `fetch_listing_format_prices` fetches HTML exactly once and passes the raw HTML to format-price parsers β no double HTTP round-trips per listing
|
| 113 |
+
- **R-PP-012:** Discovery platforms (goodreads, open_library) return `format_prices: {}` from enrichment; their single `list_price` is preserved as-is. Google Books is not a scored platform.
|
| 114 |
- **R-PP-013:** `fetch_listing_price` remains a backward-compatible thin wrapper over `fetch_listing_format_prices` returning only the primary formatted string
|
| 115 |
- **R-PP-014:** Apple Books enrichment MUST query iTunes `entity=ebook` and `entity=audiobook` so both digital formats can populate `format_prices` when available
|
| 116 |
- **R-PP-015:** Presence enrichment MUST call the same deep format parsers as source import β Amazon, BN, ThriftBooks, AbeBooks, Bookshop, Kobo β after one HTML fetch. Never invent missing format keys.
|
.agent/PRICE_INTELLIGENCE_RULES.md
CHANGED
|
@@ -59,7 +59,8 @@ No new endpoints.
|
|
| 59 |
3. No verified listings β warm text + existing `get_book_links` Buy button
|
| 60 |
|
| 61 |
**Never show as Buy / price rows in chat:** `google_books`, `goodreads`, `open_library`
|
| 62 |
-
(`BOT_CHAT_EXCLUDED_PLATFORMS`). Google Books
|
|
|
|
| 63 |
|
| 64 |
## 7. Tests
|
| 65 |
|
|
|
|
| 59 |
3. No verified listings β warm text + existing `get_book_links` Buy button
|
| 60 |
|
| 61 |
**Never show as Buy / price rows in chat:** `google_books`, `goodreads`, `open_library`
|
| 62 |
+
(`BOT_CHAT_EXCLUDED_PLATFORMS`). Google Books is also removed from Platform Presence /
|
| 63 |
+
`SCORED_PLATFORMS` β API enricher only, never a listed channel.
|
| 64 |
|
| 65 |
## 7. Tests
|
| 66 |
|
app/admin/templates/admin.html
CHANGED
|
@@ -5571,7 +5571,6 @@ const AB_PP_STATUS = {
|
|
| 5571 |
const AB_PP_CHECKLISTS = {
|
| 5572 |
amazon: ['ISBN or ASIN', 'Cover JPG (2560Γ1600 min)', 'KDP account & bank details', 'Book description & categories'],
|
| 5573 |
apple_books: ['EPUB file', 'ISBN (recommended)', 'Apple ID & iTunes Connect', 'Cover image'],
|
| 5574 |
-
google_books: ['EPUB or PDF', 'Cover image', 'Google Books partner account', 'Book metadata'],
|
| 5575 |
barnes_noble: ['EPUB file', 'ISBN', 'B&N Press account', 'Cover & description'],
|
| 5576 |
kobo: ['EPUB file', 'ISBN (recommended)', 'Kobo Writing Life account', 'Cover image'],
|
| 5577 |
bookshop: ['ISBN', 'Distributor or direct upload path', 'Book metadata'],
|
|
@@ -5579,6 +5578,7 @@ const AB_PP_CHECKLISTS = {
|
|
| 5579 |
goodreads: ['Author profile', 'Book ISBN for linking', 'Cover & description'],
|
| 5580 |
thriftbooks: ['ISBN', 'Inventory listing', 'Condition & pricing'],
|
| 5581 |
abebooks: ['ISBN', 'Seller account', 'Book condition & pricing'],
|
|
|
|
| 5582 |
};
|
| 5583 |
|
| 5584 |
let _abSupportConfig = { support_email: null };
|
|
|
|
| 5571 |
const AB_PP_CHECKLISTS = {
|
| 5572 |
amazon: ['ISBN or ASIN', 'Cover JPG (2560Γ1600 min)', 'KDP account & bank details', 'Book description & categories'],
|
| 5573 |
apple_books: ['EPUB file', 'ISBN (recommended)', 'Apple ID & iTunes Connect', 'Cover image'],
|
|
|
|
| 5574 |
barnes_noble: ['EPUB file', 'ISBN', 'B&N Press account', 'Cover & description'],
|
| 5575 |
kobo: ['EPUB file', 'ISBN (recommended)', 'Kobo Writing Life account', 'Cover image'],
|
| 5576 |
bookshop: ['ISBN', 'Distributor or direct upload path', 'Book metadata'],
|
|
|
|
| 5578 |
goodreads: ['Author profile', 'Book ISBN for linking', 'Cover & description'],
|
| 5579 |
thriftbooks: ['ISBN', 'Inventory listing', 'Condition & pricing'],
|
| 5580 |
abebooks: ['ISBN', 'Seller account', 'Book condition & pricing'],
|
| 5581 |
+
open_library: ['ISBN', 'Open Library edition record', 'Cover & description'],
|
| 5582 |
};
|
| 5583 |
|
| 5584 |
let _abSupportConfig = { support_email: null };
|
app/schemas/admin.py
CHANGED
|
@@ -239,9 +239,9 @@ class RateSessionRequest(BaseModel):
|
|
| 239 |
# βββ Publishing Help βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 240 |
|
| 241 |
_VALID_PLATFORM_IDS = frozenset({
|
| 242 |
-
"amazon", "apple_books", "
|
| 243 |
"goodreads", "thriftbooks", "abebooks", "bookshop", "open_library",
|
| 244 |
-
"
|
| 245 |
})
|
| 246 |
|
| 247 |
|
|
|
|
| 239 |
# βββ Publishing Help βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 240 |
|
| 241 |
_VALID_PLATFORM_IDS = frozenset({
|
| 242 |
+
"amazon", "apple_books", "barnes_noble", "kobo",
|
| 243 |
"goodreads", "thriftbooks", "abebooks", "bookshop", "open_library",
|
| 244 |
+
"walmart",
|
| 245 |
})
|
| 246 |
|
| 247 |
|
app/services/catalog_hub.py
CHANGED
|
@@ -36,9 +36,7 @@ VERIFIED_THRESHOLD = 0.85
|
|
| 36 |
LIKELY_THRESHOLD = 0.70
|
| 37 |
|
| 38 |
# Import platform IDs that map to scored platform IDs
|
| 39 |
-
SOURCE_PLATFORM_ALIASES: dict[str, str] = {
|
| 40 |
-
"google_play": "google_books",
|
| 41 |
-
}
|
| 42 |
|
| 43 |
|
| 44 |
@dataclass
|
|
@@ -298,14 +296,6 @@ def discover_platform_hit(
|
|
| 298 |
if url and classify_retailer_url(url, platform_id) == "product":
|
| 299 |
return PlatformHit(1.0, url, isbn_match=True)
|
| 300 |
|
| 301 |
-
if platform_id == "google_books":
|
| 302 |
-
if snap.gb_volume_id and snap.gb_confidence >= VERIFIED_THRESHOLD:
|
| 303 |
-
return PlatformHit(
|
| 304 |
-
snap.gb_confidence,
|
| 305 |
-
snap.google_books_url(),
|
| 306 |
-
snap.gb_confidence >= 0.95,
|
| 307 |
-
)
|
| 308 |
-
|
| 309 |
if platform_id == "open_library" and snap.ol_confidence >= VERIFIED_THRESHOLD:
|
| 310 |
url = snap.open_library_url()
|
| 311 |
if url:
|
|
|
|
| 36 |
LIKELY_THRESHOLD = 0.70
|
| 37 |
|
| 38 |
# Import platform IDs that map to scored platform IDs
|
| 39 |
+
SOURCE_PLATFORM_ALIASES: dict[str, str] = {}
|
|
|
|
|
|
|
| 40 |
|
| 41 |
|
| 42 |
@dataclass
|
|
|
|
| 296 |
if url and classify_retailer_url(url, platform_id) == "product":
|
| 297 |
return PlatformHit(1.0, url, isbn_match=True)
|
| 298 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 299 |
if platform_id == "open_library" and snap.ol_confidence >= VERIFIED_THRESHOLD:
|
| 300 |
url = snap.open_library_url()
|
| 301 |
if url:
|
app/services/isbn_catalog.py
CHANGED
|
@@ -487,33 +487,9 @@ async def verify_isbn_anchor(
|
|
| 487 |
snippet = (resp.text or "")[:120_000].lower()
|
| 488 |
tokens = [t for t in re.sub(r"[^\w\s]", " ", core).split() if len(t) > 3]
|
| 489 |
if tokens and sum(1 for t in tokens[:3] if t.lower() in snippet) < 2:
|
| 490 |
-
# #region agent log
|
| 491 |
-
try:
|
| 492 |
-
import time as _tv, json as _jv
|
| 493 |
-
with open("debug-7c8974.log", "a") as _fv:
|
| 494 |
-
_fv.write(_jv.dumps({"sessionId":"7c8974","hypothesisId":"A","location":"isbn_catalog.py:verify_isbn_anchor","message":"anchor_title_fail","data":{"platform":platform_id,"anchor":anchor,"final":final,"clean":clean,"found_tokens":sum(1 for t in tokens[:3] if t.lower() in snippet)},"timestamp":int(_tv.time()*1000)}) + "\n")
|
| 495 |
-
except Exception:
|
| 496 |
-
pass
|
| 497 |
-
# #endregion
|
| 498 |
return None
|
| 499 |
-
# #region agent log
|
| 500 |
-
try:
|
| 501 |
-
import time as _tv2, json as _jv2
|
| 502 |
-
with open("debug-7c8974.log", "a") as _fv2:
|
| 503 |
-
_fv2.write(_jv2.dumps({"sessionId":"7c8974","hypothesisId":"A","location":"isbn_catalog.py:verify_isbn_anchor","message":"anchor_success","data":{"platform":platform_id,"anchor":anchor,"final":final,"clean":clean,"status":resp.status_code},"timestamp":int(_tv2.time()*1000)}) + "\n")
|
| 504 |
-
except Exception:
|
| 505 |
-
pass
|
| 506 |
-
# #endregion
|
| 507 |
return clean
|
| 508 |
except Exception as exc:
|
| 509 |
-
# #region agent log
|
| 510 |
-
try:
|
| 511 |
-
import time as _tve, json as _jve
|
| 512 |
-
with open("debug-7c8974.log", "a") as _fve:
|
| 513 |
-
_fve.write(_jve.dumps({"sessionId":"7c8974","hypothesisId":"A","location":"isbn_catalog.py:verify_isbn_anchor","message":"anchor_exception","data":{"platform":platform_id,"anchor":anchor,"error":str(exc)[:200]},"timestamp":int(_tve.time()*1000)}) + "\n")
|
| 514 |
-
except Exception:
|
| 515 |
-
pass
|
| 516 |
-
# #endregion
|
| 517 |
logger.debug(
|
| 518 |
"ISBN anchor verify failed",
|
| 519 |
platform=platform_id,
|
|
|
|
| 487 |
snippet = (resp.text or "")[:120_000].lower()
|
| 488 |
tokens = [t for t in re.sub(r"[^\w\s]", " ", core).split() if len(t) > 3]
|
| 489 |
if tokens and sum(1 for t in tokens[:3] if t.lower() in snippet) < 2:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 490 |
return None
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 491 |
return clean
|
| 492 |
except Exception as exc:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 493 |
logger.debug(
|
| 494 |
"ISBN anchor verify failed",
|
| 495 |
platform=platform_id,
|
app/services/platform_presence.py
CHANGED
|
@@ -39,7 +39,7 @@ logger = structlog.get_logger(__name__)
|
|
| 39 |
|
| 40 |
_SCAN_TIMEOUT = 10.0
|
| 41 |
_MAX_CONCURRENT = 6
|
| 42 |
-
_SCAN_CACHE_PREFIX = "platform_scan:
|
| 43 |
_SCAN_CACHE_TTL = 21_600 # 6h
|
| 44 |
|
| 45 |
Status = Literal["verified", "likely", "not_found", "checking", "error", "skipped"]
|
|
@@ -51,7 +51,6 @@ PLATFORM_BENEFITS: dict[str, str] = {
|
|
| 51 |
"amazon": "Largest US ebook and print discovery channel",
|
| 52 |
"barnes_noble": "US brick-and-mortar and Nook readers",
|
| 53 |
"goodreads": "Social discovery and reader reviews",
|
| 54 |
-
"google_books": "Google Books catalog and preview reach",
|
| 55 |
"apple_books": "Reach readers on iPhone, iPad, and Mac",
|
| 56 |
"kobo": "Strong global indie and Rakuten audience",
|
| 57 |
"thriftbooks": "Secondary-market demand signal",
|
|
@@ -60,11 +59,11 @@ PLATFORM_BENEFITS: dict[str, str] = {
|
|
| 60 |
"open_library": "Free open catalog and lending discovery",
|
| 61 |
}
|
| 62 |
|
|
|
|
| 63 |
SCORED_PLATFORMS: list[dict] = [
|
| 64 |
{"platform_id": "amazon", "display_name": "Amazon", "weight": 2.0},
|
| 65 |
{"platform_id": "barnes_noble", "display_name": "Barnes & Noble", "weight": 1.0},
|
| 66 |
{"platform_id": "goodreads", "display_name": "Goodreads", "weight": 1.0},
|
| 67 |
-
{"platform_id": "google_books", "display_name": "Google Books", "weight": 1.2},
|
| 68 |
{"platform_id": "apple_books", "display_name": "Apple Books", "weight": 1.2},
|
| 69 |
{"platform_id": "kobo", "display_name": "Kobo", "weight": 1.0},
|
| 70 |
{"platform_id": "thriftbooks", "display_name": "ThriftBooks", "weight": 0.8},
|
|
@@ -302,13 +301,11 @@ class CatalogContext:
|
|
| 302 |
retailer_links: dict[str, str] = field(default_factory=dict)
|
| 303 |
|
| 304 |
|
| 305 |
-
# Goodreads BookLink name β platform_id
|
| 306 |
_BOOKLINK_PLATFORM_KEYS: list[tuple[str, tuple[str, ...]]] = [
|
| 307 |
("amazon", ("amazon",)),
|
| 308 |
("barnes_noble", ("barnes", "noble")),
|
| 309 |
("kobo", ("kobo",)),
|
| 310 |
-
("google_books", ("google books", "google play")),
|
| 311 |
-
("google_play", ("google play",)),
|
| 312 |
("apple_books", ("apple books", "ibooks")),
|
| 313 |
("bookshop", ("bookshop",)),
|
| 314 |
("thriftbooks", ("thriftbooks",)),
|
|
@@ -512,7 +509,8 @@ async def _build_catalog_context(
|
|
| 512 |
gr_links = {
|
| 513 |
pid: url
|
| 514 |
for pid, raw in parse_goodreads_book_links(page_html).items()
|
| 515 |
-
if
|
|
|
|
| 516 |
}
|
| 517 |
ctx.retailer_links = merge_retailer_links(ctx.retailer_links, gr_links)
|
| 518 |
if ctx.goodreads_url:
|
|
@@ -601,6 +599,8 @@ def presence_dict_from_db_listings(
|
|
| 601 |
platforms: list[PlatformListing] = []
|
| 602 |
for row in listings:
|
| 603 |
pid = row.platform_id
|
|
|
|
|
|
|
| 604 |
cfg = cfg_by_id.get(pid, {})
|
| 605 |
raw_fps = row.format_prices if isinstance(getattr(row, "format_prices", None), dict) else {}
|
| 606 |
platforms.append(PlatformListing(
|
|
@@ -690,10 +690,8 @@ async def _load_goodreads_hub(
|
|
| 690 |
page_html = await _fetch_goodreads_book_page(client, book_id, book_path)
|
| 691 |
if page_html:
|
| 692 |
for pid, raw in parse_goodreads_book_links(page_html).items():
|
| 693 |
-
if pid
|
| 694 |
-
|
| 695 |
-
continue
|
| 696 |
-
pid = "google_books"
|
| 697 |
clean = sanitize_platform_url(raw, pid)
|
| 698 |
if clean:
|
| 699 |
links[pid] = clean
|
|
@@ -716,10 +714,8 @@ async def _load_goodreads_hub(
|
|
| 716 |
page_html = await _fetch_goodreads_book_page(client, book_id, book_path)
|
| 717 |
if page_html:
|
| 718 |
for pid, raw in parse_goodreads_book_links(page_html).items():
|
| 719 |
-
if pid
|
| 720 |
-
|
| 721 |
-
continue
|
| 722 |
-
pid = "google_books"
|
| 723 |
clean = sanitize_platform_url(raw, pid)
|
| 724 |
if clean:
|
| 725 |
links[pid] = clean
|
|
@@ -754,15 +750,6 @@ def _finalize_listing_url(listing: PlatformListing) -> None:
|
|
| 754 |
clean = rewritten
|
| 755 |
except Exception:
|
| 756 |
pass
|
| 757 |
-
# #region agent log
|
| 758 |
-
_cls = classify_retailer_url(clean, pid) if clean else "none"
|
| 759 |
-
try:
|
| 760 |
-
import time as _t2, json as _j2
|
| 761 |
-
with open("debug-7c8974.log", "a") as _f2:
|
| 762 |
-
_f2.write(_j2.dumps({"sessionId":"7c8974","hypothesisId":"B","location":"platform_presence.py:_finalize","message":"finalize_url","data":{"pid":pid,"raw":listing.listing_url,"clean":clean,"cls":_cls},"timestamp":int(_t2.time()*1000)}) + "\n")
|
| 763 |
-
except Exception:
|
| 764 |
-
pass
|
| 765 |
-
# #endregion
|
| 766 |
if not clean or classify_retailer_url(clean, pid) != "product":
|
| 767 |
listing.listing_url = None
|
| 768 |
listing.status = "not_found"
|
|
@@ -961,16 +948,6 @@ async def scan_platform_presence(
|
|
| 961 |
author=snap.author or author,
|
| 962 |
)
|
| 963 |
|
| 964 |
-
# #region agent log
|
| 965 |
-
import time as _time
|
| 966 |
-
try:
|
| 967 |
-
with open("debug-7c8974.log", "a") as _f:
|
| 968 |
-
import json as _j
|
| 969 |
-
_f.write(_j.dumps({"sessionId":"7c8974","hypothesisId":"A-D","location":"platform_presence.py:after_enrich","message":"retailer_links_after_enrich","data":{"links":{k:v for k,v in (snap.retailer_links or {}).items()}},"timestamp":int(_time.time()*1000)}) + "\n")
|
| 970 |
-
except Exception:
|
| 971 |
-
pass
|
| 972 |
-
# #endregion
|
| 973 |
-
|
| 974 |
if not snap.asin:
|
| 975 |
resolved_asin = _extract_asin(scan_isbn or isbn)
|
| 976 |
if not resolved_asin and gr_links.get("amazon"):
|
|
|
|
| 39 |
|
| 40 |
_SCAN_TIMEOUT = 10.0
|
| 41 |
_MAX_CONCURRENT = 6
|
| 42 |
+
_SCAN_CACHE_PREFIX = "platform_scan:v14:"
|
| 43 |
_SCAN_CACHE_TTL = 21_600 # 6h
|
| 44 |
|
| 45 |
Status = Literal["verified", "likely", "not_found", "checking", "error", "skipped"]
|
|
|
|
| 51 |
"amazon": "Largest US ebook and print discovery channel",
|
| 52 |
"barnes_noble": "US brick-and-mortar and Nook readers",
|
| 53 |
"goodreads": "Social discovery and reader reviews",
|
|
|
|
| 54 |
"apple_books": "Reach readers on iPhone, iPad, and Mac",
|
| 55 |
"kobo": "Strong global indie and Rakuten audience",
|
| 56 |
"thriftbooks": "Secondary-market demand signal",
|
|
|
|
| 59 |
"open_library": "Free open catalog and lending discovery",
|
| 60 |
}
|
| 61 |
|
| 62 |
+
# Google Books is never scored / shown β API enricher only (not a retail channel).
|
| 63 |
SCORED_PLATFORMS: list[dict] = [
|
| 64 |
{"platform_id": "amazon", "display_name": "Amazon", "weight": 2.0},
|
| 65 |
{"platform_id": "barnes_noble", "display_name": "Barnes & Noble", "weight": 1.0},
|
| 66 |
{"platform_id": "goodreads", "display_name": "Goodreads", "weight": 1.0},
|
|
|
|
| 67 |
{"platform_id": "apple_books", "display_name": "Apple Books", "weight": 1.2},
|
| 68 |
{"platform_id": "kobo", "display_name": "Kobo", "weight": 1.0},
|
| 69 |
{"platform_id": "thriftbooks", "display_name": "ThriftBooks", "weight": 0.8},
|
|
|
|
| 301 |
retailer_links: dict[str, str] = field(default_factory=dict)
|
| 302 |
|
| 303 |
|
| 304 |
+
# Goodreads BookLink name β platform_id (Google Books / Play never mapped β not a scored channel)
|
| 305 |
_BOOKLINK_PLATFORM_KEYS: list[tuple[str, tuple[str, ...]]] = [
|
| 306 |
("amazon", ("amazon",)),
|
| 307 |
("barnes_noble", ("barnes", "noble")),
|
| 308 |
("kobo", ("kobo",)),
|
|
|
|
|
|
|
| 309 |
("apple_books", ("apple books", "ibooks")),
|
| 310 |
("bookshop", ("bookshop",)),
|
| 311 |
("thriftbooks", ("thriftbooks",)),
|
|
|
|
| 509 |
gr_links = {
|
| 510 |
pid: url
|
| 511 |
for pid, raw in parse_goodreads_book_links(page_html).items()
|
| 512 |
+
if pid not in ("google_play", "google_books")
|
| 513 |
+
and (url := sanitize_platform_url(raw, pid))
|
| 514 |
}
|
| 515 |
ctx.retailer_links = merge_retailer_links(ctx.retailer_links, gr_links)
|
| 516 |
if ctx.goodreads_url:
|
|
|
|
| 599 |
platforms: list[PlatformListing] = []
|
| 600 |
for row in listings:
|
| 601 |
pid = row.platform_id
|
| 602 |
+
if pid == "google_books":
|
| 603 |
+
continue
|
| 604 |
cfg = cfg_by_id.get(pid, {})
|
| 605 |
raw_fps = row.format_prices if isinstance(getattr(row, "format_prices", None), dict) else {}
|
| 606 |
platforms.append(PlatformListing(
|
|
|
|
| 690 |
page_html = await _fetch_goodreads_book_page(client, book_id, book_path)
|
| 691 |
if page_html:
|
| 692 |
for pid, raw in parse_goodreads_book_links(page_html).items():
|
| 693 |
+
if pid in ("google_play", "google_books"):
|
| 694 |
+
continue
|
|
|
|
|
|
|
| 695 |
clean = sanitize_platform_url(raw, pid)
|
| 696 |
if clean:
|
| 697 |
links[pid] = clean
|
|
|
|
| 714 |
page_html = await _fetch_goodreads_book_page(client, book_id, book_path)
|
| 715 |
if page_html:
|
| 716 |
for pid, raw in parse_goodreads_book_links(page_html).items():
|
| 717 |
+
if pid in ("google_play", "google_books"):
|
| 718 |
+
continue
|
|
|
|
|
|
|
| 719 |
clean = sanitize_platform_url(raw, pid)
|
| 720 |
if clean:
|
| 721 |
links[pid] = clean
|
|
|
|
| 750 |
clean = rewritten
|
| 751 |
except Exception:
|
| 752 |
pass
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 753 |
if not clean or classify_retailer_url(clean, pid) != "product":
|
| 754 |
listing.listing_url = None
|
| 755 |
listing.status = "not_found"
|
|
|
|
| 948 |
author=snap.author or author,
|
| 949 |
)
|
| 950 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 951 |
if not snap.asin:
|
| 952 |
resolved_asin = _extract_asin(scan_isbn or isbn)
|
| 953 |
if not resolved_asin and gr_links.get("amazon"):
|
app/services/platform_pricing.py
CHANGED
|
@@ -33,8 +33,8 @@ logger = structlog.get_logger(__name__)
|
|
| 33 |
|
| 34 |
_PRICE_TIMEOUT = 12.0
|
| 35 |
|
| 36 |
-
# Goodreads
|
| 37 |
-
_NO_PRICE_PLATFORMS = frozenset({"goodreads"})
|
| 38 |
|
| 39 |
# Discovery / enricher-only β never fetch HTML for format prices (exclude open_library which returns Free).
|
| 40 |
_NO_FORMAT_PLATFORMS = frozenset({"goodreads", "google_books"})
|
|
@@ -416,9 +416,6 @@ async def fetch_listing_price(
|
|
| 416 |
return hit
|
| 417 |
|
| 418 |
# Priority 2: built-in free logic
|
| 419 |
-
if platform_id == "google_books":
|
| 420 |
-
return await fetch_google_books_price(client, listing_url, ctx)
|
| 421 |
-
|
| 422 |
if platform_id == "apple_books":
|
| 423 |
return await fetch_apple_books_price(client, listing_url, ctx)
|
| 424 |
|
|
@@ -589,7 +586,7 @@ def _deep_format_prices_from_html(platform_id: str, html: str) -> dict[str, str]
|
|
| 589 |
|
| 590 |
def _single_price_credential_format(platform_id: str) -> str:
|
| 591 |
"""Best-effort format label for a single credential price (empty if unknown)."""
|
| 592 |
-
if platform_id
|
| 593 |
return "eBook"
|
| 594 |
# Print-focused / mixed retailers: don't guess a binding.
|
| 595 |
return ""
|
|
|
|
| 33 |
|
| 34 |
_PRICE_TIMEOUT = 12.0
|
| 35 |
|
| 36 |
+
# Goodreads / Google Books β never retail list-price channels.
|
| 37 |
+
_NO_PRICE_PLATFORMS = frozenset({"goodreads", "google_books"})
|
| 38 |
|
| 39 |
# Discovery / enricher-only β never fetch HTML for format prices (exclude open_library which returns Free).
|
| 40 |
_NO_FORMAT_PLATFORMS = frozenset({"goodreads", "google_books"})
|
|
|
|
| 416 |
return hit
|
| 417 |
|
| 418 |
# Priority 2: built-in free logic
|
|
|
|
|
|
|
|
|
|
| 419 |
if platform_id == "apple_books":
|
| 420 |
return await fetch_apple_books_price(client, listing_url, ctx)
|
| 421 |
|
|
|
|
| 586 |
|
| 587 |
def _single_price_credential_format(platform_id: str) -> str:
|
| 588 |
"""Best-effort format label for a single credential price (empty if unknown)."""
|
| 589 |
+
if platform_id == "kobo":
|
| 590 |
return "eBook"
|
| 591 |
# Print-focused / mixed retailers: don't guess a binding.
|
| 592 |
return ""
|
static/admin.html
CHANGED
|
@@ -839,7 +839,7 @@ function renderPreview(data) {
|
|
| 839 |
showAlert('error', data.error + ' β please fill in details manually.');
|
| 840 |
} else if (data.error || (data.warnings && data.warnings.length)) {
|
| 841 |
const m = data.error || data.warnings.join('; ');
|
| 842 |
-
showAlert('warning', 'β οΈ Partial data enriched via
|
| 843 |
} else { hideAlert(); }
|
| 844 |
|
| 845 |
setFetchState('preview');
|
|
|
|
| 839 |
showAlert('error', data.error + ' β please fill in details manually.');
|
| 840 |
} else if (data.error || (data.warnings && data.warnings.length)) {
|
| 841 |
const m = data.error || data.warnings.join('; ');
|
| 842 |
+
showAlert('warning', 'β οΈ Partial data enriched via catalog APIs: ' + m);
|
| 843 |
} else { hideAlert(); }
|
| 844 |
|
| 845 |
setFetchState('preview');
|
static/import-book.html
CHANGED
|
@@ -749,12 +749,11 @@
|
|
| 749 |
{ name: 'Amazon', icon: 'π' },
|
| 750 |
{ name: 'Barnes & Noble', icon: 'π' },
|
| 751 |
{ name: 'Goodreads', icon: 'β' },
|
| 752 |
-
{ name: 'Google Books', icon: 'π' },
|
| 753 |
{ name: 'Apple Books', icon: 'π' },
|
| 754 |
{ name: 'Kobo', icon: 'π' },
|
| 755 |
{ name: 'ThriftBooks', icon: 'πΈ' },
|
| 756 |
{ name: 'AbeBooks', icon: 'ποΈ' },
|
| 757 |
-
{ name: '
|
| 758 |
{ name: 'Open Library', icon: 'π' },
|
| 759 |
];
|
| 760 |
|
|
@@ -899,7 +898,7 @@
|
|
| 899 |
showAlert('error', data.error + ' β please fill in the details manually.');
|
| 900 |
} else if (data.error || (data.warnings && data.warnings.length)) {
|
| 901 |
const msg = data.error || data.warnings.join('; ');
|
| 902 |
-
showAlert('warning', 'β οΈ Partial data β some fields enriched via
|
| 903 |
} else {
|
| 904 |
hideAlert();
|
| 905 |
}
|
|
|
|
| 749 |
{ name: 'Amazon', icon: 'π' },
|
| 750 |
{ name: 'Barnes & Noble', icon: 'π' },
|
| 751 |
{ name: 'Goodreads', icon: 'β' },
|
|
|
|
| 752 |
{ name: 'Apple Books', icon: 'π' },
|
| 753 |
{ name: 'Kobo', icon: 'π' },
|
| 754 |
{ name: 'ThriftBooks', icon: 'πΈ' },
|
| 755 |
{ name: 'AbeBooks', icon: 'ποΈ' },
|
| 756 |
+
{ name: 'Bookshop.org', icon: 'π' },
|
| 757 |
{ name: 'Open Library', icon: 'π' },
|
| 758 |
];
|
| 759 |
|
|
|
|
| 898 |
showAlert('error', data.error + ' β please fill in the details manually.');
|
| 899 |
} else if (data.error || (data.warnings && data.warnings.length)) {
|
| 900 |
const msg = data.error || data.warnings.join('; ');
|
| 901 |
+
showAlert('warning', 'β οΈ Partial data β some fields enriched via catalog APIs: ' + msg);
|
| 902 |
} else {
|
| 903 |
hideAlert();
|
| 904 |
}
|
tests/bdd/features/platform_presence.feature
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
Feature: Platform Presence Scan
|
| 2 |
-
API-first discovery across
|
| 3 |
|
| 4 |
Scenario: Kindle-only Amazon book finds Amazon and Goodreads
|
| 5 |
Given a sample book "amazon_kindle"
|
|
@@ -15,8 +15,8 @@ Feature: Platform Presence Scan
|
|
| 15 |
And "open_library" should be verified with a product URL
|
| 16 |
And at least 5 platforms should be listed
|
| 17 |
|
| 18 |
-
Scenario: Platform list matches the
|
| 19 |
When I load the scored platform list
|
| 20 |
-
Then there should be exactly
|
| 21 |
-
And the platform list should include "google_books"
|
| 22 |
And the platform list should include "open_library"
|
|
|
|
| 1 |
Feature: Platform Presence Scan
|
| 2 |
+
API-first discovery across 9 public book platforms (Google Books excluded).
|
| 3 |
|
| 4 |
Scenario: Kindle-only Amazon book finds Amazon and Goodreads
|
| 5 |
Given a sample book "amazon_kindle"
|
|
|
|
| 15 |
And "open_library" should be verified with a product URL
|
| 16 |
And at least 5 platforms should be listed
|
| 17 |
|
| 18 |
+
Scenario: Platform list matches the 9 supported retailers
|
| 19 |
When I load the scored platform list
|
| 20 |
+
Then there should be exactly 9 platforms
|
| 21 |
+
And the platform list should not include "google_books"
|
| 22 |
And the platform list should include "open_library"
|
tests/bdd/steps/platform_steps.py
CHANGED
|
@@ -69,3 +69,8 @@ def step_platform_count(context, count):
|
|
| 69 |
@then('the platform list should include "{platform_id}"')
|
| 70 |
def step_platform_in_list(context, platform_id):
|
| 71 |
assert platform_id in context.platform_ids
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
@then('the platform list should include "{platform_id}"')
|
| 70 |
def step_platform_in_list(context, platform_id):
|
| 71 |
assert platform_id in context.platform_ids
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
@then('the platform list should not include "{platform_id}"')
|
| 75 |
+
def step_platform_not_in_list(context, platform_id):
|
| 76 |
+
assert platform_id not in context.platform_ids
|
tests/unit/test_catalog_hub.py
CHANGED
|
@@ -15,8 +15,9 @@ from app.services.isbn_catalog import classify_retailer_url, sanitize_platform_u
|
|
| 15 |
|
| 16 |
|
| 17 |
class TestNormalizeSourcePlatform:
|
| 18 |
-
def
|
| 19 |
-
|
|
|
|
| 20 |
|
| 21 |
def test_amazon_unchanged(self):
|
| 22 |
assert normalize_source_platform("amazon") == "amazon"
|
|
@@ -36,11 +37,11 @@ class TestDiscoverPlatformHit:
|
|
| 36 |
assert hit.confidence == 1.0
|
| 37 |
assert hit.listing_url == "https://www.amazon.com/dp/B0DHWS3P6T"
|
| 38 |
|
| 39 |
-
def
|
| 40 |
snap = self._snap(gb_volume_id="abc123", gb_confidence=0.92)
|
| 41 |
hit = discover_platform_hit("google_books", snap)
|
| 42 |
-
assert hit.listing_url
|
| 43 |
-
assert hit.confidence
|
| 44 |
|
| 45 |
def test_open_library_verified(self):
|
| 46 |
snap = self._snap(isbn13="9780593128251", ol_confidence=0.90, ol_key="/works/OL123W")
|
|
|
|
| 15 |
|
| 16 |
|
| 17 |
class TestNormalizeSourcePlatform:
|
| 18 |
+
def test_google_play_not_aliased_to_scored_platform(self):
|
| 19 |
+
# Google Books is not a scored channel; Play stays as itself.
|
| 20 |
+
assert normalize_source_platform("google_play") == "google_play"
|
| 21 |
|
| 22 |
def test_amazon_unchanged(self):
|
| 23 |
assert normalize_source_platform("amazon") == "amazon"
|
|
|
|
| 37 |
assert hit.confidence == 1.0
|
| 38 |
assert hit.listing_url == "https://www.amazon.com/dp/B0DHWS3P6T"
|
| 39 |
|
| 40 |
+
def test_google_books_never_discovered_as_listing(self):
|
| 41 |
snap = self._snap(gb_volume_id="abc123", gb_confidence=0.92)
|
| 42 |
hit = discover_platform_hit("google_books", snap)
|
| 43 |
+
assert hit.listing_url is None
|
| 44 |
+
assert hit.confidence == 0.0
|
| 45 |
|
| 46 |
def test_open_library_verified(self):
|
| 47 |
snap = self._snap(isbn13="9780593128251", ol_confidence=0.90, ol_key="/works/OL123W")
|
tests/unit/test_platform_presence.py
CHANGED
|
@@ -298,9 +298,9 @@ async def test_build_catalog_uses_slug_fetch_and_bookshop_isbn():
|
|
| 298 |
|
| 299 |
|
| 300 |
@pytest.mark.asyncio
|
| 301 |
-
async def
|
| 302 |
from app.services.platform_presence import scan_cache_key
|
| 303 |
-
assert scan_cache_key("https://example.com/book").startswith("platform_scan:
|
| 304 |
|
| 305 |
|
| 306 |
@pytest.mark.asyncio
|
|
@@ -341,11 +341,12 @@ async def test_scan_mocks_catalog_hub():
|
|
| 341 |
author="Jane Author",
|
| 342 |
isbn="9780306406157",
|
| 343 |
)
|
| 344 |
-
assert len(SCORED_PLATFORMS) ==
|
| 345 |
verified_ids = {p.platform_id for p in scan.platforms if p.status == "verified"}
|
| 346 |
-
assert "google_books" in verified_ids
|
| 347 |
assert "goodreads" in verified_ids
|
| 348 |
assert "open_library" in verified_ids
|
|
|
|
| 349 |
|
| 350 |
|
| 351 |
def test_replace_from_scan_defaults_visible_in_widget_false():
|
|
|
|
| 298 |
|
| 299 |
|
| 300 |
@pytest.mark.asyncio
|
| 301 |
+
async def test_scan_cache_key_busted_for_v14():
|
| 302 |
from app.services.platform_presence import scan_cache_key
|
| 303 |
+
assert scan_cache_key("https://example.com/book").startswith("platform_scan:v14:")
|
| 304 |
|
| 305 |
|
| 306 |
@pytest.mark.asyncio
|
|
|
|
| 341 |
author="Jane Author",
|
| 342 |
isbn="9780306406157",
|
| 343 |
)
|
| 344 |
+
assert len(SCORED_PLATFORMS) == 9
|
| 345 |
verified_ids = {p.platform_id for p in scan.platforms if p.status == "verified"}
|
| 346 |
+
assert "google_books" not in verified_ids
|
| 347 |
assert "goodreads" in verified_ids
|
| 348 |
assert "open_library" in verified_ids
|
| 349 |
+
assert "google_books" not in {p["platform_id"] for p in SCORED_PLATFORMS}
|
| 350 |
|
| 351 |
|
| 352 |
def test_replace_from_scan_defaults_visible_in_widget_false():
|