"""connectors_tt.py — the TIKTOK connector (wave 29 · item 7 · DEBT D-9 · rulings R1 + R2). Everything in this file knows what a VENDOR's TikTok row looks like. Nothing in it knows what an automation is. That split is `connectors_ig.py`'s (wave 27 item 23) and it is the reason this file exists at all rather than another thousand lines inside the engine. ⛔ **EVERY VENDOR FIELD NAME HERE WAS PROBED, NOT GUESSED.** The whole schema — 40 profile / 43 post / 17 comment fields, each with the vendor's own type, description and `pii` flag — was read live from `GET /datasets/{id}/metadata` for **$0.00** and written down in `.claude/wiki/waves/wave29/proto/tiktok-schema.md` (promoted to `tiktok-capture.md` at close-out). That document is the AUTHORITY: do not re-probe it, and do not invent a key. Where a name below reads through a candidate list it is because the vendor has two names for one fact (`biography`/`signature`, `region`/`country`), never because the name is uncertain. ⛔ **NOTHING HERE EVER AUTHENTICATES TO TIKTOK.** No login, no cookie, no account to get banned — public data through a supplier, exactly the rail `connectors_ig.py` states for Instagram. The vendor key is a key to a SUPPLIER. ⭐ **THE TRANSPORT IS `connectors_bd.py` — SHARED, VENDOR-NAMED, AND NO LONGER BORROWED FROM THE OTHER PLATFORM'S CONNECTOR** (WAVE 30 · T09, DEBT D-128). `bd_call`, `bd_scrape` and `bd_filter_start` take the dataset id as a PARAMETER — they are Bright Data's wire, not Instagram's — and re-implementing them here would be a second copy of the deferral handling, the truncation guard, the SSRF rail and the snapshot-progress reader, i.e. five places for one bug. Until wave 30 the code was right and the NAME was wrong: this file imported thirteen symbols from `connectors_ig`, which read as a dependency on Instagram and was really a dependency on a supplier. ⚠ **This file now imports ZERO names from `connectors_ig`, and a gate check asserts that**, because the sentence above is the kind that quietly stops being true. ⚠ **WHAT $0 COULD NOT BUY, so nobody reads this file as more measured than it is:** 1. the real ROW shape — `/metadata` describes a DATASET, and Instagram's rows carry undeclared envelope keys (`timestamp`, `input`) that no metadata call mentions; 2. that a declared field POPULATES — Bright Data's Instagram Reels *declares* `views: number` and delivers an account-grain wrong number (§4e). **Declared is not delivered**, and the one TikTok claim that matters most (`play_count`) is exactly a declaration. """ from __future__ import annotations import automation_engine as engine # ⚠ T09 — FOUR NAMES CAME OFF THIS LIST AND NOTHING BROKE, which is the point of deriving an # import block from the AST both ways. `bd_call`, `bd_filter_start`, `bd_key` and `bd_ready` were # imported here under a comment claiming they were *"re-exported for the runners"*; no runner ever # read them off this module (the engine imports them from the transport itself), so they were four # lines of dependency nobody was paying for. `[[artifact-with-no-importer]]` in its smallest form. from connectors_bd import ( _bd_first_url, _bd_flag, _bd_list, _bd_source_payload, _first, _ig_int, bd_scrape, ) # --------------------------------------------------------------------------------------------- # THE DATASETS # --------------------------------------------------------------------------------------------- # ⚠ CATALOGUE PRESENCE IS NOT ENTITLEMENT — the same finding Instagram produced. `GET # /datasets/list` returned 1,735 rows of which 12 are TikTok; the three below answer 200 with a # full field list, and the two DISCOVERY halves answer **404 for our key**: # `gd_lj71gn6l68bz7y9hc` (posts by profile) and `gd_lilwhto81z415d9mdl` (posts by keyword). # ⇒ TikTok discovery routes through the PROFILES dataset's corpus filter, exactly as Instagram's # does. A ticket that reaches for a by-keyword endpoint is reaching for a 404. TT_DS_PROFILES = "gd_l1villgoiiidt09ci" # TikTok - Profiles. 40 fields, 152,000,000 records TT_DS_POSTS = "gd_lu702nij2f790tmv9h" # TikTok - Posts. 43 fields TT_DS_COMMENTS = "gd_lkf2st302ap89utw5k" # TikTok - Comments. 17 fields #: The vendor's two post-type tokens, verbatim from the dataset's own `ai_description` #: (*"strictly these two"*, video = 99.5% of rows). Ours are `image`/`video`/`carousel`. TT_POST_TYPE_VIDEO = "video" TT_POST_TYPE_CONTENT = "content" #: What a TikTok profile URL looks like, for the runner that has a handle and needs a URL. Kept #: beside the dataset ids because it is the same class of vendor fact. #: ⚠ ONE SPELLING. `platform/core/user_tables.profile_url(handle, 'tiktok')` builds the identical #: string from `_PROFILE_RULES` (contract C2, E's half) — this exists for the connector's own #: batch calls, and the gate asserts the two agree rather than trusting that they do. TT_PROFILE_URL = "https://www.tiktok.com/@{handle}" def tt_profile_url(handle): """`nurilab` → `https://www.tiktok.com/@nurilab`. `''` for a blank handle, never a bare `@`.""" h = str(handle or "").strip().lstrip("@") return TT_PROFILE_URL.format(handle=h) if h else "" # --------------------------------------------------------------------------------------------- # THE FIELD MAPS # --------------------------------------------------------------------------------------------- # Each function turns ONE vendor row into the cell dict for one of the `ut_tt_*` schemas declared # in `automation_engine`. The rules they all obey, stated once: # # * **BLANK MEANS NOT READ, NEVER "THEY HAVE NONE".** A key the vendor did not send is OMITTED, # so a later, richer pull fills it instead of being overwritten by this one's silence. `_first` # returns `None` (never 0) when nothing matches, which is what makes that possible. # * **A zero from the vendor is a MEASUREMENT and survives.** (Instagram's `_ig_zero_is_blank` # rule is scoped to a paid rung whose zeros were proven fictional; nothing here has earned it.) # * **Every unpromoted vendor key stays whole in `source_payload`.** A schema addition on the # vendor's side is preserved rather than silently discarded while our column model catches up. # * **Nothing here writes a session token.** `tt_chain_token`, `secu_id` (~85% null), `short_id` # (100% null in the sample), `ftc` (100% null) and `relation` are deliberately unmapped; they # ride in `source_payload` where they make no claim. def _tt_str(node, *names): """The first non-empty string among `names`, or None when the vendor sent nothing. ⛔ `None`, NOT `""`. The callers below drop `None` keys, which is what keeps a blank honest — an empty string written into a cell claims "we looked and it is empty". """ v = _first(node, *names) if v is None: return None s = str(v).strip() return s or None def _tt_pct(node, *names): """A vendor 0–1 engagement fraction → our stored 0–100 percentage, or None. ⛔ THE ×100 IS NOT COSMETIC (wave 26, amendment C1-a). Our `pct` renderer appends the sign to the STORED number, so writing the vendor's raw 0.0656 would print a 6.6% creator as `0.0%` — measured on the Instagram side, and TikTok sends the same shape on all three of its rates. """ v = _first(node, *names) if v is None: return None out = engine._pct100(v) return out or None def _tt_day(node, *names): """A vendor stamp → `YYYY-MM-DD`, or None. Our `date` columns store a day.""" v = _first(node, *names) if v is None: return None return engine._day(v) or None def _drop_blanks(row): """The one place a mapped row loses its `None`s — see the BLANK MEANS NOT READ rule above.""" return {k: v for k, v in row.items() if v is not None and v != ""} def normalize_profile(node, handle=""): """A TikTok Profiles row → the `ut_tt_profile` / `ut_tt_snapshots` cell shape. ⚠ TWO FIELDS READ THROUGH A CANDIDATE PAIR, and both pairs are the vendor's, not a guess: * `biography` is PRIMARY and `signature` the FALLBACK — the probe measured `signature` populated on 85% of rows and they carry the same text; * `region` is PRIMARY and `country` the FALLBACK — `region` is the one with a documented two-letter-ISO description, `country` has no description at all. ⚠ `videos_count` is mapped to `posts_count` with a caveat recorded rather than hidden: its `ai_description` ranges 1-89, so it may be a WINDOW rather than a lifetime total. It is the only count of its kind the dataset offers. """ node = node if isinstance(node, dict) else {} account = _tt_str(node, "account_id") or str(handle or "").strip().lstrip("@") return _drop_blanks({ "platform": engine.PLATFORM_TIKTOK, "handle": account, "full_name": _tt_str(node, "nickname"), "tt_id": _tt_str(node, "id"), "profile_url": _tt_str(node, "url") or (tt_profile_url(account) or None), "bio": _tt_str(node, "biography", "signature"), "external_url": _bd_first_url(_first(node, "bio_link")), "verified": _bd_flag(node, "is_verified"), "is_private": _bd_flag(node, "is_private"), # ⚠ APPROXIMATE, and the word is the vendor's: `is_commerce_user` has *"many null values"* # on their own description. It is the closest thing TikTok has to Instagram's # `is_business_account`, and `_bd_flag` writes nothing at all when the key is absent — so # the approximation only ever fills a cell the vendor actually answered. "is_business": _bd_flag(node, "is_commerce_user"), "followers": _ig_int(_first(node, "followers")), "following": _ig_int(_first(node, "following")), "posts_count": _ig_int(_first(node, "videos_count")), # ⚠ `likes` on a PROFILE row is likes RECEIVED across the account's videos (18-110,200, no # nulls). It is not a post-level number and it is not our `likes` column, which is why it # is stored under a different name. "likes_received": _ig_int(_first(node, "likes")), "avg_engagement": _tt_pct(node, "awg_engagement_rate"), "like_engagement": _tt_pct(node, "like_engagement_rate"), "comment_engagement": _tt_pct(node, "comment_engagement_rate"), "country_code": _tt_str(node, "region", "country"), "region": _tt_str(node, "region"), "predicted_lang": _tt_str(node, "predicted_lang"), # ⚠ ACCOUNT AGE, NOT A MEASUREMENT STAMP — `create_time` on a profile is when the ACCOUNT # was made. TikTok stamps nothing with "when this number was true", exactly like Instagram, # which is why the append law dates a snapshot by when WE read it. "account_created_at": _tt_day(node, "create_time"), "source_payload": _bd_source_payload(node), }) def tt_post_type(node): """A TikTok post row → one of OUR three type options, or None. ⛔ THE FIRST OF THE PROBE DOC'S TWO NAMED BLOCKERS. The vendor's vocabulary is `"video"` / `"content"`; ours is `image` / `video` / `carousel` and has no `"content"`. Writing the vendor's token would fail `_clean_field`'s option check on the way in and would put an untranslated API word in front of a user on the way out. So: `video` is `video`, and `content` — TikTok's photo-mode post — is `image`, EXCEPT when the row carries more than one `carousel_images` entry, which is what a carousel IS on either network. ⚠ The multi-image branch is decided from the IMAGES, never from the type token: the token cannot express it, so inferring `carousel` from the word would be inventing a fact. ⚠ An UNKNOWN token returns None rather than defaulting to `video` (99.5% of rows are video, and that is exactly what would make the wrong default invisible). """ raw = str((node or {}).get("post_type") or "").strip().lower() if raw == TT_POST_TYPE_VIDEO: return "video" if raw != TT_POST_TYPE_CONTENT: return None images = (node or {}).get("carousel_images") return "carousel" if isinstance(images, list) and len(images) > 1 else "image" def normalize_post(node): """A TikTok Posts row → the `ut_tt_posts` cell shape. None when it carries no identity. ⛔ THE SECOND NAMED BLOCKER, RESOLVED HERE AND NOWHERE ELSE: `play_count` is ONE number and Instagram's schema has TWO columns for it (`plays` and `views`). On TikTok they are the same fact — `play_count` IS the count TikTok displays under a video — so it maps to `views` and `ut_tt_posts` HAS NO `plays` COLUMN. Copying one vendor number into two of our columns would manufacture a second measurement that a rollup could average or double-count, which is a worse outcome than the missing column it would paper over. ⚠ `shortcode` reads `shortcode` then `post_id`: both are 19-digit numerics on this dataset and the probe measured them as the same shape. That equality is what lets the comments→posts link join with NO normaliser, which the Instagram side never had. ⚠ `num_share_count` (a number) is preferred over `share_count` (typed TEXT by the vendor). ⚠ `commerce_info` is a business/commerce LOCATION per its own description — cities and countries. It is NOT a paid-partnership flag, and nothing in this dataset is: TikTok declares no equivalent, so `paid_partnership`/`partner` have no column on this family at all. """ node = node if isinstance(node, dict) else {} shortcode = _tt_str(node, "shortcode", "post_id") if not shortcode: return None return _drop_blanks({ "platform": engine.PLATFORM_TIKTOK, "shortcode": shortcode, # ⛔⛔ `account_id`, NOT `profile_username` — MEASURED on a real Posts row 2026-08-12. # The vendor's `profile_username` is the DISPLAY NAME (`"Dina"`), while `account_id` is the # @handle (`"d1na_th"`) — the same field `normalize_profile` already reads for `handle`, so # one name means one thing across both corpora. Reading the display name silently broke the # only join this table has: `ut_tt_posts.influencer_key` -> `ut_tt_profile.handle` matched # NOTHING, so a person could not filter posts by creator and a rollup would count zero. # ⚠ NO FALLBACK TO `profile_username`, deliberately. It is not a degraded handle, it is a # different fact, and filling a join key with it is worse than leaving it blank — a blank # is visibly missing, a display name looks like an answer [[one-question-two-normalizers]]. # The URL is the honest second source: it carries the handle by construction. "influencer_key": (_tt_str(node, "account_id") or tt_handle(_tt_str(node, "profile_url", "url") or "") or None), "posted_at": _tt_day(node, "create_time"), "type": tt_post_type(node), "caption": _tt_str(node, "description"), "url": _tt_str(node, "url"), "hashtags": _bd_list(node, "hashtags"), "tagged_location": _tt_str(node, "commerce_info"), "views": _ig_int(_first(node, "play_count")), "likes": _ig_int(_first(node, "digg_count")), "comments": _ig_int(_first(node, "comment_count")), "shares": _ig_int(_first(node, "num_share_count")), "saves": _ig_int(_first(node, "collect_count")), "video_duration": _ig_int(_first(node, "video_duration")), "source_payload": _bd_source_payload(node), }) def normalize_comment(node): """A TikTok Comments row → the `ut_tt_comments` cell shape. None without a comment id. ⚠ THE NAME COLLISION, RESOLVED: TikTok's `replies` is an ARRAY of reply objects and OUR `replies` column is an INT count. The count comes from `num_replies`; the array stays whole in `source_payload`. Reading the array's length instead would be a second, disagreeing answer to a question the vendor already answers — and it would disagree, because a page of replies is not all of them. ⚠ `date_created` is typed `date` by this vendor, unlike Instagram's `comment_date` which is text and needs defensive parsing. It still goes through `_tt_day` — one date path, so a vendor that changes its mind cannot change ours. ⚠ The comment TEXT and every identifiable commenter field (`commenter_user_name` is flagged PII) stay in `source_payload` and are promoted to no column, which is the same posture the Instagram comment schema takes for the same D-24 reason. """ node = node if isinstance(node, dict) else {} comment_key = _tt_str(node, "comment_id") if not comment_key: return None return _drop_blanks({ "platform": engine.PLATFORM_TIKTOK, "comment_key": comment_key, "shortcode": _tt_str(node, "post_id"), # ⭐⭐ OWNER RULING 2026-08-12: the comment's CONTENT gets a column. `comment_text` is the # vendor's own key and `comment_text_only` its stripped variant (the probe recorded both); # primary first, so a row carrying the rich form is not silently served the plain one. # ⛔ The commenter's identity is deliberately NOT promoted — see `TT_COMMENT_FIELDS`. "text": _tt_str(node, "comment_text", "comment_text_only"), "commented_at": _tt_day(node, "date_created"), "likes": _ig_int(_first(node, "num_likes")), "replies": _ig_int(_first(node, "num_replies")), "source_payload": _bd_source_payload(node), }) #: ⭐ The three maps, addressable by name — so a gate (and the runners in T04-T06) can walk them #: rather than naming three functions, and so adding a fourth dataset is one entry. TT_NORMALIZERS = { "tt_profile": normalize_profile, "tt_post_metrics": normalize_post, "tt_comments": normalize_comment, } # --------------------------------------------------------------------------------------------- # THE FETCH — wave 30 · W30-T08 (carrying wave-29's dropped T05) # --------------------------------------------------------------------------------------------- def tt_handle(url): """A TikTok profile URL **or** a bare handle → the handle. `''` when it is neither. Deliberately permissive about the input and strict about the output, because the two callers hand it different things: an automation stores whatever a person typed in the profile column (`@nurilab`, `nurilab`, or the full URL), while the discovery runner already holds a clean `account_id`. One normaliser, so a row found by discovery and a row typed by hand cannot resolve to two different handles. """ s = str(url or "").strip() if not s: return "" if "tiktok.com" in s.lower(): # Everything after the first `@`, up to the next path segment or query. tail = s.split("@", 1)[1] if "@" in s else "" s = tail.split("/")[0].split("?")[0].split("#")[0] s = s.strip().lstrip("@").strip() # A handle is the vendor's `account_id` shape: alphanumerics, dots and underscores. return s if s and all(c.isalnum() or c in "._" for c in s) else "" def tt_post_urls(node, limit=0): """⭐ WAVE 30 · T10 — the profile row's own post permalinks, newest-first as the vendor sends. ⛔ THIS IS WHY TIKTOK POST CAPTURE COSTS NO EXTRA DISCOVERY. `top_videos` rides the PROFILE row we have already bought, so the posts read is a scrape of links we hold, never a search for them. The two TikTok DISCOVERY datasets (posts-by-profile, posts-by-keyword) are **404 for our key**, so a design that reached for either would not merely be dearer, it would not work. ⛔⛔ CORRECTED 2026-08-12 — THIS DOCSTRING USED TO SAY *"the probe MEASURED `top_videos` as an array of video permalinks, NO empties"*, AND THAT SENTENCE IS WHAT SHIPPED THE BUG. The probe read `/datasets/{id}/metadata` — a DATASET description — and the phrase quoted was the field's `ai_description`, not an observation of a row. The real row sends **dicts keyed `video_url`** (measured below), so the reader built against the quoted sentence found nothing, forever, in silence. ⭐ The transferable half: *"the probe measured X"* and *"the probe read a declaration of X"* are different claims, and prose cannot be told apart by a reader downstream — which is why the correction names the method, not just the value. ⚠ `top_posts_data` is deliberately NOT read: the probe calls it *"a thin dup of `top_videos`"*, and preferring whichever happened to be longer is how one creator's window silently differs from another's. ⚠ `limit <= 0` means "everything the row carried". The CAP IS THE CALLER'S — `config.maxPosts`, validated 1..12 — and it is applied here rather than after the scrape so an unwanted post is never bought. [[a-constant-two-features-share]]: the 12 is the vendor's measured profile window, not a number this function may invent. """ raw = (node or {}).get("top_videos") out = [] for item in raw if isinstance(raw, list) else []: # ⛔⛔ MEASURED ON A REAL ROW 2026-08-12, AND IT IS NOT WHAT THE SCHEMA SAID. # `top_videos` is NOT an array of permalink strings. One paid TikTok Profiles scrape of a # live handle returns **19 DICTS**, keyed # `video_url · video_id · playcount · diggcount · commentcount · share_count · # favorites_count · create_date · cover_image`. # The docstring above cites the $0 probe as having "measured" permalinks — it had not, and # could not: `/datasets/{id}/metadata` describes a DATASET, and the probe's own verdict says # so in terms (*"what $0 cannot buy … the real ROW shape … that a declared field # POPULATES"*, `wave29/proto/tiktok-schema.md`). This is the SECOND time this vendor's # declaration has diverged from its delivery on this exact axis; BD's IG Reels `views` was # the first. [[reachable-is-not-the-same-as-built]] # ⇒ The consequence, live: `TT_DS_POSTS` was never reached, because this returned an EMPTY # list on every real profile — post capture could not have worked for anybody, and the # T10 gate stayed green because its canned fixture encoded the DECLARED shape. A fixture # written from a schema tests the schema. # ⚠ `video_url` is FIRST because it is the key the vendor actually sends; `url` is kept # because it costs nothing and is what a future corpus revision would most likely use. The # bare-string branch stays for the same reason — this widens what is ACCEPTED and invents # nothing: a shape that yields no `http…` value still degrades to "no posts", exactly as # before, rather than to a URL built out of a guess. # ⚠ `top_posts_data` is STILL not read (it carries `post_url` and would work): preferring # whichever array happened to be longer is how one creator's window silently differs from # another's, and that reasoning is unchanged by this correction. if isinstance(item, dict): u = str(item.get("video_url") or item.get("url") or "").strip() else: u = str(item or "").strip() if u.lower().startswith("http") and u not in out: out.append(u) return out[:limit] if limit and limit > 0 else out def pull_posts_tt(post_urls, log=print, deferred=None): """The TikTok Posts dataset for a list of permalinks → `(rows, note)`, already normalised. ⚠ ONE CALL FOR THE WHOLE WINDOW. `bd_scrape` has always taken a list, and the Instagram side measured what happens when a caller forgets: 25 records, one billed snapshot each, a walk still running at 67 minutes. Nothing here loops per URL. """ urls = [str(u) for u in (post_urls or []) if str(u or "").strip()] if not urls: return [], "" rows, note = bd_scrape(TT_DS_POSTS, urls, deferred=deferred) if note: log(f"[aios-tt] posts: {note}") return [], note out = [r for r in (normalize_post(n) for n in rows) if r] return out, "" def pull_comments_tt(post_urls, log=print, deferred=None): """The TikTok Comments dataset for a list of POST permalinks → `(rows, note)`, normalised. ⛔ THE MOST EXPENSIVE THING THIS PRODUCT BUYS, and the reason `commentMetrics` defaults OFF on both networks: a comments scrape ingests identifiable third parties who never entered anybody's list (D-24). The mapper already keeps every commenter field in `source_payload` and promotes none of them to a column; this function adds no new exposure, it just has to be asked for. """ urls = [str(u) for u in (post_urls or []) if str(u or "").strip()] if not urls: return [], "" rows, note = bd_scrape(TT_DS_COMMENTS, urls, deferred=deferred) if note: log(f"[aios-tt] comments: {note}") return [], note out = [r for r in (normalize_comment(n) for n in rows) if r] return out, "" #: ⭐⭐ WAVE 30 · D-156 — THE MEDIA DATASETS, AS A SET, SO THE HAND-OFF CAN FILTER ON IDENTITY. #: `_media_deferrals` uses this to lift ONLY posts/comments snapshots out of the local deferral #: list. That is what makes it structurally impossible to file a PROFILE snapshot in the engine's #: metric queue — the defect a draft of T10 shipped and A-39 booked as "the wrong fix is worse #: than the gap". A membership test cannot be got wrong by a later edit the way `if` order can. TT_MEDIA_DATASETS = (TT_DS_POSTS, TT_DS_COMMENTS) def _media_deferrals(deferred): """The POSTS/COMMENTS entries of a `bd_scrape` deferral list — never the profile's. ⚠ The engine, not this module, decides what a deferral MEANS: it stamps `kind` and the handle and files it. TikTok needs no `_tag_metric_deferrals` twin because one dataset is one kind here, so the id already carries everything a mapper choice depends on — and importing Instagram's tagger is not available anyway (W30-T09 gates ZERO `from connectors_ig` lines). """ out = [] for d in deferred or []: if isinstance(d, dict) and str(d.get("datasetId") or "") in TT_MEDIA_DATASETS: out.append(dict(d)) return out def pull_profile_tt(url, log=print, pending_profile=None, prefetch=None, max_posts=0, post_metrics=False, comment_metrics=False): """ONE TikTok profile from the vendor. Same return contract as `pull_profile`. `{state, profile, posts, comments, via, note}` with `state ∈ ok | partial | blocked | error`, so the engine's enrich branch treats every network identically and no caller learns a new shape. ⭐ WAVE 30 · T10 — POSTS AND COMMENTS ARE REAL NOW, AND BOTH DEFAULT OFF, exactly as Instagram's do. `post_metrics` scrapes the profile row's own `top_videos` permalinks (see `tt_post_urls` — no discovery call, because both TikTok discovery datasets 404 for our key); `comment_metrics` then scrapes the comments of the posts that came back. ⚠ COMMENTS REQUIRE POSTS by construction rather than by a rule: their input IS a post permalink, so asking for comments with post capture off is a request with no subject, and it returns none instead of quietly buying posts nobody asked for. ⛔ `partial` IS THE SUCCESS STATE WHENEVER NO MEDIA WAS READ, and that is deliberate rather than pessimistic. The Instagram contract reads `ok` only when identity AND media both landed (`pull_profile_bd`: *"identity without media is still partial ... a run that wrote a follower count and no posts must not paint green over a posts table that did not grow"*). So: posts not ASKED for → `partial`, saying so; posts asked for and landed → `ok`; asked for and none came → `partial` with the vendor's reason. The state answers "did this pull deliver what it went for", never "did the function finish". ⚠ **NO FREE RUNG, AND NO FALLBACK CHAIN.** Instagram's `pull_profile` drops to Apify when the paid rung refuses; `providers.DEFAULT_CHAINS["tt_profile"]` is deliberately single-provider, with its own note explaining that a multi-provider chain is a promise something walks it and that nothing walks Instagram's second name today either. So a refusal here is final, and it says so instead of implying a retry somewhere. """ handle = tt_handle(url) if not handle: return {"state": "error", "profile": {}, "posts": [], "comments": [], "via": "", "note": f"{url!r} is not a TikTok profile URL or handle"} # ⭐ THE BATCH FAST PATH, same shape as the Instagram side: `prefetch` is `{handle: node}` from # one multi-URL scrape covering a whole selection. A hit is a vendor round trip that does not # happen; a miss falls through to the single-URL call below. cached = prefetch.get(handle) if isinstance(prefetch, dict) else None _deferred = [] if isinstance(cached, dict) and cached: rows, note = [cached], "" else: rows, note = bd_scrape(TT_DS_PROFILES, [tt_profile_url(handle)], deferred=_deferred) node = rows[0] if rows else {} profile = normalize_profile(node, handle) if node else {} # ⛔ THE READABILITY TEST IS `followers`/`following`, NOT "did we get a dict". `normalize_profile` # drops blanks, so an unreadable row still returns `{"platform": …, "handle": …}` — truthy, and # carrying nothing anybody asked for. The Instagram rung tests exactly this pair for exactly # this reason, and answering "0 followers" instead is the failure it exists to prevent. unreadable = profile.get("followers") is None and profile.get("following") is None if note or unreadable: # ⭐ THE DEFERRAL IS HANDED OVER RATHER THAN DISCARDED. A snapshot the vendor is still # building HAS ALREADY BEEN PAID FOR; dropping its id bills again on the next run for the # same record. That was live on the Instagram profile path until 2026-08-09 — measured on # nurilab as two runs, two fresh snapshots, both abandoned — and it is not being # reintroduced here by omission. if isinstance(pending_profile, list): for d in _deferred: pending_profile.append({**d, "kind": "profile", "influencer": handle}) why = note or ("the scrape answered, but no follower/following counts were readable in it " "(the field names may have moved - see tiktok-capture.md)") return {"state": "blocked", "profile": {}, "posts": [], "comments": [], "via": "brightdata", "deferredProfile": [d.get("snapshotId") for d in _deferred], "note": why} # --- W30-T10: THE MEDIA, ONLY WHEN IT WAS ASKED FOR. ------------------------------------ if not post_metrics: return {"state": "partial", "profile": profile, "posts": [], "comments": [], "via": "brightdata", "note": note or "profile read; post capture is off for this step"} urls = tt_post_urls(node, limit=max_posts) if not urls: # ⚠ NOT AN ERROR AND NOT A RETRY. A creator with no `top_videos` has nothing to buy, and # saying so is what stops the next run paying to be told the same thing. return {"state": "partial", "profile": profile, "posts": [], "comments": [], "via": "brightdata", "note": note or "profile read; this account's row carried no post links"} posts, p_note = pull_posts_tt(urls, log=log, deferred=_deferred) comments, c_note = ([], "") if comment_metrics and posts: # The comments dataset is keyed on a POST permalink, so it reads the posts we just bought — # `url` from the mapper, never the profile's raw array, so a post the posts scrape refused # is not silently asked about again one rung later. comments, c_note = pull_comments_tt([p.get("url") for p in posts if p.get("url")], log=log, deferred=_deferred) # ⭐⭐ WAVE 30 · D-156 — THE MEDIA DEFERRALS ARE HANDED BACK, and the shape of the hand-off is # the whole lesson. An earlier draft of T10 appended every `_deferred` entry to # `pending_profile` tagged `kind: "profile"`. By the time control reaches here a PROFILE # deferral is impossible — the profile branch above returns `blocked` on any note — so **every # id fanned out that way was a POSTS or COMMENTS snapshot in the PROFILE queue**, whose # collector writes preset profile cells onto somebody's record from post rows. The engine keeps # the two queues apart deliberately (`_pending_profile_tasks` vs `_pending_metric_tasks`). # ⇒ So this returns them under their OWN key, filtered by dataset identity # (`_media_deferrals`), and the engine files them in the metric queue with the handle it # already holds. Returning rather than appending also keeps the queue's vocabulary out of a # connector: this module knows which CORPUS deferred, never what the engine calls it. # ⚠ `deferredMedia` rides BOTH returns on purpose. The empty-posts case is the one that # matters most — that is exactly the run where the vendor took too long, so a caller reading # the ids only from the success path would lose every batch it actually paid for. deferred_media = _media_deferrals(_deferred) if not posts: return {"state": "partial", "profile": profile, "posts": [], "comments": [], "via": "brightdata", "deferredMedia": deferred_media, "note": p_note or note or "profile read; the post source returned nothing"} return {"state": "ok", "profile": profile, "posts": posts, "comments": comments, "via": "brightdata", "deferredMedia": deferred_media, "note": c_note or note or ""}