Hugging Face's logo

Spaces:
MSGEncrypted
/
lesson-agent-dev
Sleeping

App Files Community
lesson-agent-dev / skills /scrape-web /references /html-cleanup.md
MSG
Feat/research tab agent skills (#5)
e7fd66f
preview code
|
Raw
History Blame Contribute Delete
317 Bytes

A newer version of the Gradio SDK is available: 6.20.0

Upgrade

HTML cleanup

ResearchMind uses trafilatura to strip boilerplate and keep main article text.

  • include_tables=true for data-heavy pages
  • include_comments=false
  • Fallback: first 50k chars of raw HTML if extraction returns empty

Raw snapshot saved under RESEARCHMIND_DATA_DIR/raw/{doc_id}/snapshot.txt.