docs/README.md

Rescored Documentation

Project Vision

Rescored is an AI-powered music transcription and notation editor that converts YouTube videos into editable sheet music. Musicians can paste a URL, get professional-quality notation, edit it, and export in standard formats.

What This Documentation Covers

This documentation serves as the technical blueprint for implementing Rescored. It focuses on:

• Architecture & Design: System structure, technology choices, and rationale
• Backend Processing: Audio extraction, ML transcription pipeline, API design
• Frontend Interface: Sheet music rendering, interactive editing, playback
• Integration: How components communicate and data flows through the system
• Implementation Roadmap: MVP scope and future feature phases

Reading Guide

For Full-Stack Developers (Start Here)

1. Getting Started - Overview and context
2. Architecture Overview - How everything fits together
3. MVP Scope - What to build first
4. Deep dive into Backend and Frontend sections

For Backend Engineers

1. Architecture Overview
2. Technology Stack - Backend choices
3. Audio Processing Pipeline - Core workflow
4. API Design
5. Background Workers
6. Testing Guide - Writing and running tests

For Frontend Engineers

1. Architecture Overview
2. Technology Stack - Frontend choices
3. Notation Rendering
4. Interactive Editor
5. Playback System
6. Data Flow

For Product/Design

1. MVP Scope
2. Architecture Overview - User flow
3. Known Challenges - Limitations to design around

Documentation Structure

Architecture

• System Overview - High-level architecture and data flow
• Technology Stack - Tech choices with alternatives and trade-offs
• Deployment Strategy - Infrastructure and scaling

Backend

• Audio Processing Pipeline - End-to-end audio → notation workflow
• API Design - REST endpoints and WebSocket protocol
• Background Workers - Async job processing with Celery
• Testing Guide - Backend test suite and best practices

Frontend

• Notation Rendering - Sheet music display with VexFlow
• Interactive Editor - Editing operations and state management
• Playback System - Audio synthesis with Tone.js
• Data Flow - State management and API integration

Integration

• File Formats - MusicXML, MIDI, internal JSON
• WebSocket Protocol - Real-time progress updates

Features

• MVP Scope - Phase 1 features and future roadmap
• Instrument Remapping - Cross-instrument conversion (future)

Research

• ML Model Selection - Model comparison and benchmarks
• Technical Challenges - Known limitations and edge cases

Reference

• Getting Started - How to use this documentation
• Glossary - Musical and technical terminology

Key Principles

1. MVP First: Focus on single-instrument (piano) transcription before multi-instrument
2. Quality Over Speed: Prioritize transcription accuracy over processing time
3. Editable Output: Transcription won't be perfect—editor is critical for fixing errors
4. Standard Formats: Use MusicXML/MIDI for maximum compatibility
5. Async Everything: Audio processing is slow—use queues and WebSocket updates

Quick Reference

Primary Use Case: YouTube URL → Transcribed piano sheet music → Edit → Export

Tech Stack Summary:

• Frontend: React + VexFlow + Tone.js
• Backend: Python/FastAPI + Celery + Redis
• ML: Demucs (source separation) + basic-pitch (transcription)
• Formats: MusicXML (primary), MIDI (export)

MVP Timeline: Focused on getting piano transcription working end-to-end with basic editing

Contributing to Documentation

As implementation progresses, update these docs with:

• Actual code examples and API samples
• Performance benchmarks and metrics
• Lessons learned and gotchas
• Configuration details and environment setup

Need Help?

• See Glossary for terminology
• Check Challenges for known issues
• Review Tech Stack for decision context