feat: Add validation to skip duplicate header rows in batch processing

chore: Update ignored files and refine prompt content.

feat: enhance text cleaning utility to remove HTML tags and entities, update documentation, and requirements.

Clean deployment without large index files