fix: robots-txt not loaded error conditions

docs: minor fix

docs: be neutral on brand

fix: block suspicious requests before sideload

feat: control concurrent request per page instead of server bucket

fix: unhandledRejection log level

chore: prefer ctx.URL

chore: add comments to clarify

fix: missing url query param

fix: finalizer and unhandled promise rejection

fix: robots-txt location

fix: robots-txt cache location

fix: curl redirections

chore: fix search result amount (#1163)

fix: curl error category

fix: bad snapshot in sideload should not throw directly

feat: leveraging NODE_COMPILE_CACHE (#1162)

fix: several crash cases

feat: md options pass though to turndown

fix: catch jsdom errors

fix: guard invalid domain names

fix: potential fix for firestore grpc connection reset

fix

fix: somehow side-loading chromewebstore would 100% crash the browser

fix: expect failure while loading pdf

fix: potential invalid pdf issue

cleanup: use local project code as much as possible

fix: sideload redirections

fix: bug in pptr injections

fix: fail early on special cookie redirects

fix: curl cookie mimicking

fix: curl cookie behavior

fix: implement DNT in alt-gen and pdf-extract

fix: sideLoad header detection

fix: curl redirection location

fix: curl implicit redirect

fix: side load context bridging

fix: clean HTML timer

fix: remove tailwind classes instead of the opposite

fix: curl error wrapping

restructure: nolonger a firebase application (#1160)