feat: improve JSON parsing and error handling by introducing isLikelyTruncatedJson function to detect incomplete JSON responses. Update client logic to handle truncation retries more effectively and enhance error messaging for truncated responses. Add tests to validate new functionality and ensure robust error reporting.

feat: refactor AI response handling by introducing parseAiJsonResponse and related utilities to improve JSON parsing and error handling. Streamline response processing in client and pipeline modules, enhancing robustness against HTML error pages and reasoning blocks. Add tests for new parsing functions to ensure reliability.

feat: add db:seed command to package.json and update README.md for seeding reference data. Enhance job status handling in generation logic to support completed_partial status, improving error reporting and user feedback. Introduce new utility functions for job outcome resolution and non-retryable error detection, along with corresponding tests. Update database migration scripts to include new tables and fields for improved data management.