app/project.txt

Knowledge Base: Studio-Based Product Scraping Bot for AdultDVDMarketplace
1. Project Overview
This project is about building a browser automation and data extraction bot for a website where products are listed under different studios. The user wants a tool that can automatically log in, navigate to a chosen studio, sort the products by price, and scrape all products whose price is equal to or greater than a user-defined minimum price.
The extracted result should include at least:
•	Product title
•	UPC
•	Optional price
•	Optional product URL
•	Optional studio name
This is not a normal static website scraper. It is a login-based, multi-step browser automation workflow. That means the bot must behave like a user in a browser, click through menus, navigate listings, open products, and collect details.
This project sits in the category of:
•	browser automation
•	web scraping
•	data extraction
•	workflow automation
•	reusable scraping tool
________________________________________
2. What the Client Wants
The client described the required behavior very clearly.
They want a bot where they can provide:
•	a studio URL or studio name
•	a minimum price threshold
The bot should then:
1.	Open the target website
2.	Pass the age gate / warning page
3.	Log in using valid account credentials
4.	Navigate to the studio section
5.	Open the specific studio’s video listing
6.	Sort products by price from lowest to highest
7.	Skip all products below the user’s chosen minimum price
8.	Start scraping from the first product that matches the threshold
9.	Continue scraping all qualifying products up to the highest price
10.	Extract the required fields, especially:
•	title
•	UPC
11.	Save the results into a file such as CSV or Excel
The client also indicated that:
•	the website allows bot usage
•	login credentials will be shared later
•	the screenshots and video were provided to explain the steps
________________________________________
3. What This Project Really Is
From a technical and delivery standpoint, this is not just “a crawler.”
It is a stateful automated browser bot with structured scraping logic.
That means it has to handle:
•	authentication
•	session persistence
•	menu-based navigation
•	price sorting
•	pagination
•	item-by-item extraction
•	detail-page drilling for UPC
•	output file generation
So the correct framing of the project is:
Build a reusable automation bot that logs into a website, navigates to a selected studio, sorts products by ascending price, applies a minimum price filter, extracts title and UPC from all matching products, and exports the results into a structured file.
________________________________________
4. Business Goal of the Project
The business value is simple:
The client does not want to manually browse every studio, sort items, open products one by one, and copy title and UPC details. That is time-consuming and error-prone.
The bot will eliminate manual repetitive work by:
•	reducing time spent on research
•	increasing consistency
•	improving extraction speed
•	creating reusable datasets
•	enabling repeat runs for different studios and price thresholds
This means the project is not only about scraping. It is about operational efficiency.
________________________________________
5. High-Level Workflow
The whole bot workflow can be understood in 7 major phases.
Phase 1: Access Website
The bot opens the target website.
Phase 2: Pass Entry/Age Gate
The site shows a warning/age confirmation page. The bot must click Enter before proceeding.
Phase 3: Login
The bot signs into the user’s account using provided credentials.
Phase 4: Navigate to Studio
The bot goes to the Top Studios area and selects the required studio by URL or name.
Phase 5: Open Product Listing
The bot opens the studio’s DVD/video listing page.
Phase 6: Sort and Filter
The bot sorts products by price ascending and starts scraping only from the first product with price greater than or equal to the user’s minimum price.
Phase 7: Extract and Save
The bot collects product details and exports them into CSV/Excel.
________________________________________
6. Detailed Website Flow Based on the Screenshots
From the screenshots, the workflow appears to be:
Step A — Warning Page
The site first shows an age/consent warning page with an ENTER button and an EXIT button.
Expected bot action:
•	detect the warning screen
•	click ENTER
Step B — Login Page / Logged-In Sidebar
The next screenshots show the site with a login form and later an account panel indicating the user is logged in.
Expected bot action:
•	locate username field
•	locate password field
•	submit login
•	confirm successful account login
Step C — Top Studios Menu
The screenshots show a top menu item called TOP STUDIOS, which opens a dropdown menu with featured studios and a VIEW ALL STUDIOS option.
Expected bot action:
•	hover or click TOP STUDIOS
•	either:
o	choose VIEW ALL STUDIOS
o	or go directly to the studio page if a URL is provided
Step D — Studios Directory
The all-studios page shows alphabet-based filtering and a studio list.
Expected bot action if studio name is used:
•	navigate to correct letter section
•	find the target studio
•	click the “DVD Movies” entry for that studio
Step E — Studio Product List
The next screenshots show a specific studio page like 20/20 Vision DVD Movies.
Expected bot action:
•	confirm studio page opened correctly
•	identify product cards / rows
•	locate sort options
Step F — Sort by Price
The page has sort options such as:
•	popularity
•	title
•	price
•	release date
Expected bot action:
•	click Price
•	verify results are sorted low to high
Step G — Product Cards
Each product entry shows:
•	title
•	price
•	image
•	buy button
•	short description
Expected bot action:
•	read visible product title and price
•	compare price to user minimum threshold
•	open each qualifying product detail page
Step H — Product Detail Page
The detail page contains metadata including:
•	format
•	studio
•	release date
•	SKU
•	UPC
•	category
Expected bot action:
•	extract UPC
•	extract title
•	optionally extract studio, price, release date, SKU, category, URL
•	return to listing or proceed to next item
________________________________________
7. Functional Requirements
These are the must-have capabilities of the system.
7.1 Input Requirements
The tool must accept:
•	Studio URL or Studio Name
•	Minimum price threshold
•	Login credentials
•	Output format preference
Optional inputs:
•	include price in output
•	include URL in output
•	include SKU in output
•	include release date in output
•	maximum number of items to scrape
•	whether to run headless or visible browser
________________________________________
7.2 Core Functional Requirements
Authentication
•	must pass age gate
•	must log in successfully
•	must maintain session
Navigation
•	must locate studio by URL or name
•	must open studio’s DVD/video listing
Sorting
•	must sort products by price ascending
Filtering
•	must skip products below threshold
•	must scrape products at threshold and above
Extraction
For each matching item, the bot must capture:
•	Title
•	UPC
Recommended additional fields:
•	Price
•	Product URL
•	Studio Name
•	SKU
•	Release Date
•	Category
Pagination
•	must continue to all pages, if multiple pages exist
Export
•	must save extracted results in structured format
________________________________________
8. Non-Functional Requirements
These are quality requirements, not just feature requirements.
Stability
The bot should survive small delays and page load variations.
Accuracy
The bot should correctly read titles, prices, and UPC values.
Reusability
The tool should not be hardcoded for one run only. It should work repeatedly for different studios and prices.
Maintainability
The code should be modular so future changes are easier.
Security
Credentials must not be hardcoded in public code or logs.
Transparency
The bot should log what it is doing so issues are easy to debug.
________________________________________
9. Why This Cannot Be Built as a Simple Scraper
A lot of beginners think scraping means using requests and BeautifulSoup. That is not enough here.
This project requires browser automation because the site involves:
•	age gate
•	login-protected sections
•	interactive menus
•	dynamic navigation
•	detail-page clicking
•	sorting behavior that may be driven by the browser session
So this should be built with tools such as:
•	Playwright preferred
•	Selenium acceptable alternative
Playwright is stronger because it is:
•	faster
•	more modern
•	better at waiting for elements
•	better for dynamic websites
•	cleaner for multi-step automation
________________________________________
10. Recommended Technical Stack
Best Stack
•	Python
•	Playwright
•	Pandas
•	CSV / Excel export
Supporting Libraries
•	playwright
•	pandas
•	openpyxl for Excel output
•	logging
•	dataclasses or structured classes for clean architecture
Why Python
•	easier automation ecosystem
•	clean scripting
•	strong export handling
•	quick delivery
•	easier future maintenance
________________________________________
11. Suggested Bot Architecture
A clean implementation should be split into modules.
Module 1: Config / Inputs
Handles:
•	studio input
•	price threshold
•	credentials
•	output format
•	run settings
Module 2: Browser Session Manager
Handles:
•	launch browser
•	open page
•	manage cookies/session
•	close browser
Module 3: Authentication Handler
Handles:
•	age gate
•	login
•	login validation
Module 4: Studio Navigator
Handles:
•	finding studio by URL
•	finding studio by name
•	navigating to listing page
Module 5: Listing Scraper
Handles:
•	sorting by price
•	scanning product cards
•	threshold logic
•	page traversal
Module 6: Product Detail Scraper
Handles:
•	opening detail page
•	extracting UPC and other metadata
•	returning structured record
Module 7: Export Manager
Handles:
•	CSV export
•	Excel export
•	deduplication
•	file naming
Module 8: Logger / Error Handler
Handles:
•	progress logging
•	retry handling
•	failure reporting
________________________________________
12. Core Business Logic in Plain English
This is the heart of the project.
The site contains product listings for a specific studio. Those listings can be sorted by price. The user does not want everything. The user only wants products starting from a chosen minimum price.
So the logic is:
1.	Sort by lowest price first
2.	Read products in ascending order
3.	Ignore products cheaper than the user’s threshold
4.	Once the first product at or above the threshold is found:
o	scrape it
o	scrape every product after it
5.	Continue until no more products remain
This is efficient because sorted data reduces unnecessary extraction work.
________________________________________
13. Exact Step-by-Step Process the Bot Must Perform
This section is the most important for anyone who needs to understand the project end-to-end.
Step 1 — Start Process
Bot receives:
•	studio name or URL
•	minimum price
•	login credentials
Step 2 — Open Website
Bot launches browser and goes to the website homepage.
Step 3 — Accept Warning Page
Bot identifies the warning/age page and clicks ENTER.
Step 4 — Login
Bot locates the username and password fields, enters credentials, and signs in.
Step 5 — Confirm Login Worked
Bot checks that login was successful by looking for account panel, account username, logout button, or account-related menu.
Step 6 — Navigate to the Studio
Two possible flows:
Flow A: Studio URL is provided
•	bot directly opens the studio page
Flow B: Studio Name is provided
•	bot goes to Top Studios
•	opens all studios
•	finds matching studio
•	clicks relevant studio DVD Movies listing
Step 7 — Confirm Studio Listing Page
Bot verifies the current page belongs to the correct studio.
Step 8 — Sort by Price
Bot clicks the Price sorting option and waits for the sorted listing.
Step 9 — Scan Visible Products
For each visible listing:
•	read title
•	read price
•	compare price against threshold
Step 10 — Threshold Decision
•	if price is below threshold → skip
•	if price is at/above threshold → open and scrape
Step 11 — Open Product Detail
For each qualifying product:
•	click product
•	extract title and UPC
•	optionally extract price, SKU, release date, category, URL
Step 12 — Save Record in Memory
Bot stores each scraped result in a list or dataframe.
Step 13 — Return to Listing
Bot returns to listing page and proceeds with the next product.
Step 14 — Move Through Pagination
When current page is done:
•	detect next page or continue loading
•	repeat scraping process
Step 15 — End Condition
Stop when:
•	there are no more products
•	or optional max limit is reached
Step 16 — Export Results
Write all records to CSV or Excel.
Step 17 — Final Report
Bot outputs:
•	total products checked
•	total products scraped
•	output file path
•	any skipped/error items
________________________________________
14. Data Fields That Can Be Scraped
Minimum Required Fields
•	Title
•	UPC
Recommended Fields
•	Price
•	Studio Name
•	Product URL
•	SKU
•	Release Date
•	Category
•	Description snippet
Best-Practice Output Schema
A robust output file should contain columns like:
•	studio_name
•	title
•	upc
•	price
•	sku
•	release_date
•	category
•	product_url
•	scraped_at
This turns the bot into a more scalable data tool.
________________________________________
15. Inputs Needed From the Client
Before the project can be built properly, these are the required client-side inputs.
Mandatory
•	Website URL
•	Test login credentials
•	Example studio name or URL
•	Minimum price example
•	Desired output format
Strongly Recommended
•	Confirmation that automation is allowed
•	Sample expected output file
•	Whether this is one-time use or reusable tool
•	Whether they need visible browser or headless mode
•	Whether they need only Title + UPC or more fields
________________________________________
16. Questions You Should Ask the Client
These are the right operational questions to lock the scope.
About Access
1.	What is the exact website URL?
2.	Please share test credentials for login.
3.	Is there any IP restriction, captcha, or 2FA on login?
4.	Is the age-gate shown every session or only on first visit?
About Input
5.	Will the bot always receive a studio URL, or sometimes only a studio name?
6.	If studio name is provided, should it match exact name only or partial match too?
7.	Can studio names have duplicates or similar variations?
About Filtering
8.	Should the bot scrape products priced equal to the threshold, or strictly greater than it?
9.	Is price always in USD?
10.	Should the bot include new price, used price, or whichever appears on listing?
About Data Extraction
11.	Do you only need title and UPC, or also price, SKU, release date, product link, and category?
12.	If UPC is missing on some products, should that record be skipped or included with blank UPC?
13.	Should duplicate products be removed from output?
About Pagination and Coverage
14.	Should the bot scrape all pages until the end?
15.	Is there ever a need to stop after a fixed number of products?
About Output
16.	What output format do you want: CSV, Excel, JSON, or Google Sheet?
17.	Do you want one file per studio or one merged file for all runs?
18.	What should the output filename format be?
About Use Case
19.	Is this a one-time scrape or a reusable tool?
20.	Do you need a command-line script, desktop tool, or simple UI dashboard?
21.	Should non-technical users be able to run it?
About Deployment
22.	Should it run on your local machine, a VPS, or cloud server?
23.	On which operating system should it run?
24.	Do you need setup documentation and support after delivery?
________________________________________
17. Things That Must Be Clarified Before Final Price or Timeline
These variables directly affect effort and cost.
•	Is login simple or protected?
•	Is studio navigation straightforward or inconsistent?
•	Is pagination normal or infinite scroll?
•	Is UPC always on product detail page?
•	Is there a lot of page delay or lazy loading?
•	Does sorting actually work reliably from the UI?
•	Are there duplicate or broken product pages?
•	How polished should the tool be:
o	script only
o	reusable CLI
o	full interface
Without these clarified, any estimate is soft.
________________________________________
18. Risks and Technical Challenges
Even if the site allows bots, there are still execution risks.
Challenge 1: Login Stability
Login selectors can change. Session handling must be solid.
Challenge 2: Dynamic Menus
Dropdown menus like “Top Studios” may require hover and timing control.
Challenge 3: Sorting Validation
Clicking “Price” does not automatically guarantee true ascending order. The bot may need to validate.
Challenge 4: Product Detail Dependency
UPC appears on the detail page, not necessarily in listing cards. This means more page visits and slower scraping.
Challenge 5: Pagination
The page may have standard pagination, lazy loading, or stateful URLs.
Challenge 6: Inconsistent Price Fields
Some pages may show:
•	lowest price
•	new price
•	used price
The project must define which price matters.
Challenge 7: Missing UPCs or Broken Pages
The bot should handle incomplete records gracefully.
________________________________________
19. Recommended Rules for Error Handling
A professional bot should not just crash on one issue.
It should handle:
•	missing title
•	missing price
•	missing UPC
•	product page not opening
•	next page not loading
•	session expiring
•	unexpected redirects
Recommended behavior:
•	log the error
•	skip or retry
•	continue the run
•	show summary at the end
________________________________________
20. Success Criteria
The project should be considered successful if:
1.	User can input studio and minimum price
2.	Bot logs in successfully
3.	Bot reaches the correct studio page
4.	Bot sorts by ascending price
5.	Bot scrapes all products at/above threshold
6.	Bot extracts title and UPC correctly
7.	Bot saves output in requested format
8.	Bot can be rerun with different studios and prices
________________________________________
21. Out-of-Scope Items Unless Explicitly Requested
These should not be assumed unless the client asks.
•	bypassing captchas or anti-bot systems
•	parallel scraping across many studios
•	database integration
•	cloud deployment
•	web dashboard
•	Google Sheets live sync
•	proxy rotation
•	schedule-based automation
•	multi-account rotation
These can be future upgrades.
________________________________________
22. Recommended Deliverables
A professional delivery package should include:
Core Deliverable
•	working automation script/tool
Additional Deliverables
•	requirements file
•	installation instructions
•	usage guide
•	sample output file
•	configuration instructions
•	source code with comments
•	error/logging support
•	short video or screenshots of working flow
________________________________________
23. Best Delivery Version Options
There are three sensible ways to package this project.
Option 1: Basic Script
User edits values in code and runs script.
Best for:
•	technical user
•	low budget
•	one-person usage
Option 2: Config-Based Tool
User updates values in config file or command line.
Best for:
•	reusable internal tool
•	moderate budget
•	practical production use
Option 3: Simple UI Tool
User enters studio and min price in a small interface and clicks Run.
Best for:
•	non-technical user
•	repeat usage
•	premium version
________________________________________
24. Recommended Internal Workflow for Building the Project
If you or a developer is actually implementing this, this is the clean execution plan.
Stage 1 — Discovery
•	confirm URL
•	confirm selectors
•	test login
•	test studio navigation
•	inspect product detail page
Stage 2 — MVP Build
•	open site
•	pass age gate
•	log in
•	navigate studio
•	sort price
•	scrape title + UPC
•	export CSV
Stage 3 — Stabilization
•	add retries
•	add logs
•	improve waits
•	handle pagination
•	handle missing data
Stage 4 — Packaging
•	config support
•	Excel export
•	documentation
•	final testing
________________________________________
25. Testing Plan
The project should be tested against these scenarios:
Test Case 1
Input studio URL + min price = 5
Expected: only products with price >= 5 extracted
Test Case 2
Input studio name instead of URL
Expected: bot finds correct studio and continues
Test Case 3
Threshold lower than all products
Expected: all products extracted
Test Case 4
Threshold higher than all products
Expected: empty file or no-match message
Test Case 5
Missing UPC on one product
Expected: script handles it without crashing
Test Case 6
Multi-page listing
Expected: bot continues across all pages
________________________________________
26. Plain-English Summary for a Beginner
If someone knows nothing about the project, explain it like this:
This project is a smart browser bot that logs into a website and automatically collects product information from a selected studio. The user tells it which studio to inspect and the minimum price they care about. The bot then goes through the studio’s product list, sorts everything by price, skips cheap products, and collects the title and UPC of all products from that price upward. Finally, it saves everything into a spreadsheet file.
That is the simplest accurate summary.
________________________________________
27. Short Professional Scope Statement
You can use this as a formal scope description:
Develop a reusable browser automation bot that logs into the target website, navigates to a specified studio by URL or name, sorts studio products by ascending price, filters products based on a user-defined minimum price threshold, extracts product title and UPC from qualifying items, handles pagination, and exports the results in CSV or Excel format.
________________________________________
28. What You Should Tell Someone Building It
Give them this exact operational brief:
•	Use Python with Playwright
•	Build modular code
•	Do not hardcode selectors everywhere
•	Handle age gate and login first
•	Support studio URL and studio name input
•	Sort by price ascending
•	Start scraping only when price reaches threshold
•	Open product detail pages for UPC
•	Handle pagination
•	Export structured results
•	Add logs and retries
•	Keep credentials secure
________________________________________
29. Final Recommendation
This should be positioned as a reusable automation utility, not a one-off disposable script.
That matters because the workflow is structured enough to become a repeatable internal tool. The better commercial angle is:
•	scalable
•	reusable
•	configurable
•	low manual effort
•	high operational value
So the strongest implementation path is:
Python + Playwright + CSV/Excel export + configurable inputs + proper logs
________________________________________
30. Final One-Paragraph Master Explanation
This project is a login-based browser automation bot for scraping studio-specific DVD product data from a website. The user supplies a studio name or URL and a minimum price threshold. The bot opens the website, passes the age confirmation page, logs into the account, navigates to the selected studio’s product listing, sorts the products by price from lowest to highest, skips all products below the threshold, opens each qualifying product page, extracts the title and UPC, and saves the results into a structured output file such as CSV or Excel. The project requires browser automation rather than simple scraping because it involves protected access, dynamic navigation, interactive sorting, and detail-page extraction.