| # TREA 2.0 - Technical Documentation | |
| Comprehensive technical documentation for the TREA 2.0 audio dataset generation pipeline. This document covers the complete implementation including algorithms, mathematical formulations, configuration parameters, preprocessing details, and capacity-aware balancing mechanisms. | |
| **For Quick Start Guide**: See [README.md](README.md) | |
| --- | |
| ## Table of Contents | |
| 1. [Pipeline Overview](#pipeline-overview) | |
| 2. [How Sample Durations Are Generated](#how-sample-durations-are-generated) | |
| 3. [Configuration Reference](#configuration-reference) | |
| 4. [ESC-50 Preprocessing](#esc-50-preprocessing-duration-task-only) | |
| 5. [Audio Utilities](#audio-utilities) | |
| 6. [Task: COUNT](#task-count) | |
| 7. [Task: DURATION](#task-duration) | |
| 8. [Task: ORDER](#task-order) | |
| 9. [Task: VOLUME](#task-volume) | |
| 10. [Deterministic Balancing Mechanisms](#deterministic-balancing-mechanisms) | |
| 11. [Rejection Logic and Retry Mechanisms](#rejection-logic-and-retry-mechanisms) | |
| 12. [Command-Line Arguments](#command-line-arguments) | |
| 13. [Summary](#summary) | |
| --- | |
| ## Pipeline Overview | |
| ### Architecture | |
| The pipeline generates four types of audio-based question-answering samples: | |
| | Task | Question Type | Example Question | | |
| |------|---------------|------------------| | |
| | **COUNT** | Counting unique sounds | "How many unique sounds do you hear?" | | |
| | **DURATION** | Temporal comparison | "Which sound plays for the longest duration?" | | |
| | **ORDER** | Temporal ordering | "Which sound plays first/last/after X?" | | |
| | **VOLUME** | Loudness comparison | "Which sound is the loudest/softest?" | | |
| ### Directory Structure | |
| ``` | |
| pipeline/ | |
| ├── main.py # Entry point - orchestrates all tasks | |
| ├── config.yaml # All configuration parameters | |
| ├── tasks/ | |
| │ ├── task_count.py # CountTaskGenerator class | |
| │ ├── task_duration.py # DurationTaskGenerator class | |
| │ ├── task_order.py # OrderTaskGenerator class | |
| │ └── task_volume.py # VolumeTaskGenerator class | |
| ├── utils/ | |
| │ ├── __init__.py # Exports all utilities | |
| │ ├── audio_utils.py # Audio processing functions | |
| │ ├── dataset_utils.py # ESC50Dataset, PreprocessedESC50Dataset | |
| │ ├── question_utils.py # QuestionGenerator | |
| │ ├── llm_utils.py # LLMQuestionGenerator | |
| │ └── logger.py # setup_logger | |
| └── output/ # Generated outputs | |
| ``` | |
| ### Data Flow | |
| ``` | |
| ESC-50 Dataset (2000 clips, 50 categories, 5s each) | |
| ↓ | |
| [DURATION TASK ONLY] Preprocessing Script (preprocess_esc50.py) | |
| ├── Detects sound regions using adaptive noise-floor thresholding | |
| ├── Trims leading/trailing silence (keeps internal structure) | |
| ├── Calculates effective durations | |
| ↓ | |
| ESC-50_preprocessed/ | |
| ├── effective_durations.csv (metadata with effective durations) | |
| └── trimmed_audio/*.wav (edge-trimmed clips) | |
| ↓ | |
| Pipeline (task-specific generation with balancing) | |
| ├── COUNT: Uses raw ESC-50 clips | |
| ├── DURATION: Uses preprocessed clips with effective durations | |
| ├── ORDER: Uses raw ESC-50 clips | |
| └── VOLUME: Uses raw ESC-50 clips (normalized then volume-adjusted) | |
| ↓ | |
| output/{task}/ | |
| ├── audios/*.wav (generated audio samples) | |
| ├── {task}_mcq.csv (multiple choice questions) | |
| ├── {task}_open_text.csv (open-ended questions) | |
| └── {task}_metadata.csv (detailed metadata) | |
| ``` | |
| ### Entry Point: `main.py` | |
| The main orchestration happens via individual task runner functions: | |
| ```python | |
| def run_count_task(config: dict, logger): | |
| generator = CountTaskGenerator(config, logger) | |
| generator.dataset.reset_category_usage() | |
| generator.generate_dataset() | |
| def run_duration_task(config: dict, logger): | |
| generator = DurationTaskGenerator(config, logger) | |
| generator.dataset.reset_category_usage() | |
| generator.generate_dataset() | |
| def run_order_task(config: dict, logger): | |
| generator = OrderTaskGenerator(config, logger) | |
| generator.dataset.reset_category_usage() | |
| generator.generate_dataset() | |
| def run_volume_task(config: dict, logger): | |
| generator = VolumeTaskGenerator(config, logger) | |
| generator.dataset.reset_category_usage() | |
| generator.generate_dataset() | |
| ``` | |
| --- | |
| ## How Sample Durations Are Generated | |
| **IMPORTANT**: Sample durations are generated upfront to **exactly fill the target task duration**. | |
| ### The Algorithm | |
| Located in `utils/audio_utils.py`: | |
| ```python | |
| def generate_sample_durations_for_task( | |
| task_duration_hours: float, | |
| min_clip_duration: float, | |
| max_clip_duration: float | |
| ) -> list: | |
| """ | |
| Generate sample durations that exactly fill the target task duration. | |
| """ | |
| task_duration_seconds = task_duration_hours * 3600 | |
| remaining = task_duration_seconds | |
| durations = [] | |
| while remaining >= min_clip_duration: | |
| # Cap max at remaining to avoid overshoot | |
| effective_max = min(max_clip_duration, remaining) | |
| # If remaining is less than min, we can't fit another sample | |
| if effective_max < min_clip_duration: | |
| break | |
| # Sample uniformly within valid range | |
| d = random.uniform(min_clip_duration, effective_max) | |
| durations.append(d) | |
| remaining -= d | |
| # Shuffle to randomize order | |
| random.shuffle(durations) | |
| return durations | |
| ``` | |
| 1. Start with `remaining = total_seconds` | |
| 2. While `remaining >= min_clip_duration`: | |
| - Sample `d ~ Uniform(min, min(max, remaining))` | |
| - Append `d` to durations list | |
| - Subtract `d` from remaining | |
| 3. Shuffle and return | |
| ### Mathematical Properties | |
| **Guarantee**: $\sum_{i=1}^{N} d_i \leq T$ and $T - \sum d_i < d_{\min}$ | |
| Where: | |
| - $T$ = total task duration | |
| - $d_i$ = duration of sample $i$ | |
| - $d_{\min}$ = minimum clip duration | |
| - $N$ = number of samples generated (variable, not fixed!) | |
| **Each duration**: $d_i \sim \text{Uniform}(d_{\min}, \min(d_{\max}, \text{remaining}_i))$ | |
| ### Example | |
| With `task_duration_size = 1.0` hours (3600s), `min = 20s`, `max = 60s`: | |
| ``` | |
| remaining=3600 → d₁=45.2s → remaining=3554.8 | |
| remaining=3554.8 → d₂=28.7s → remaining=3526.1 | |
| remaining=3526.1 → d₃=52.1s → remaining=3474.0 | |
| ... | |
| remaining=35.2 → d₈₉=35.2s → remaining=0 (capped at remaining) | |
| ``` | |
| Result: 89 samples totaling exactly 3600s (instead of estimated 90) | |
| ### Where It's Called | |
| Each task's `generate_dataset()` method uses this: | |
| ```python | |
| def generate_dataset(self) -> tuple: | |
| # Generate all durations upfront | |
| sample_durations = generate_sample_durations_for_task( | |
| self.task_duration_hours, | |
| self.min_clip_duration, | |
| self.max_clip_duration | |
| ) | |
| num_samples = len(sample_durations) | |
| self.logger.info(f"Generating {num_samples} samples...") | |
| # Each sample uses its pre-assigned duration | |
| for i, target_duration in enumerate(sample_durations): | |
| metadata = self.generate_sample(i, target_duration=target_duration, ...) | |
| ``` | |
| ``` | |
| --- | |
| ## Configuration Reference | |
| All parameters are defined in `config.yaml`. | |
| ### Dataset Class Subset Configuration | |
| ```yaml | |
| dataset: | |
| use_class_subset: false # Enable to use only a subset of ESC-50 classes | |
| num_classes_subset: 40 # Number of classes for train/val/test (e.g., 40 of 50) | |
| subset_persist_path: "output/class_subset.json" # Path to save/load class subset | |
| subset_seed: 42 # Random seed for subset selection (persisted) | |
| ``` | |
| **Purpose**: Create in-distribution (ID) splits using a subset of classes, then optionally test on out-of-distribution (OOD) using all classes. | |
| **Workflow**: | |
| 1. Set `use_class_subset: true` and `num_classes_subset: 40` | |
| 2. Run pipeline - 40 classes randomly selected and saved to `class_subset.json` | |
| 3. Generate train/val/test splits - all use same 40 classes | |
| 4. For OOD test: Set `use_class_subset: false`, use different output path | |
| ### Global Audio Parameters | |
| ```yaml | |
| audio: | |
| min_clip_duration: 20.0 # Minimum generated clip duration (seconds) | |
| max_clip_duration: 60.0 # Maximum generated clip duration (seconds) | |
| source_clip_duration: 5.0 # ESC-50 clip length (seconds) | |
| # Silence and crossfade parameters (applied to ALL tasks) | |
| min_silence_duration: 100 # Minimum silence ALWAYS between clips (ms) | |
| max_extra_silence_per_gap: 500 # Max extra silence per gap when distributing remainder (ms) | |
| crossfade_duration: 500 # Crossfade between audio-silence transitions (ms) for smooth joins | |
| crossfade_within_source: 50 # Small crossfade within same-source repetitions (ms) for COUNT task | |
| with_silence: true # Enable silence insertion between clips | |
| normalize: false | |
| normalize_target_dBFS: -20.0 | |
| ``` | |
| ### Task-Specific Parameters | |
| #### COUNT Task | |
| ```yaml | |
| count: | |
| enabled: true | |
| task_duration_size: 2.0 # Hours of total audio to generate | |
| max_clips_per_sample: 10 # Maximum unique sounds per sample (1 to 10) | |
| ordering_mode: "random" # "random" (shuffled clips) or "consecutive" (grouped by source) | |
| # CAPACITY-AWARE ANSWER BALANCING: | |
| # - Creates balanced distribution of answers from 1 to max_clips_per_sample | |
| # - Sorts samples by capacity (max_clips each can fit) | |
| # - Assigns higher targets to high-capacity samples | |
| # - Clamps targets to what actually fits (reduces excessive silence) | |
| ``` | |
| #### DURATION Task | |
| ```yaml | |
| duration: | |
| enabled: true | |
| task_duration_size: 2.0 | |
| preprocessed_data_path: "/home/debarpanb1/TREA_2.0/ESC-50_preprocessed" | |
| question_types: ["shortest", "longest"] | |
| num_unique_sources: 10 # Can be int or list (e.g., [2,3,4,5]) | |
| ordering_methods: ["consecutive"] # Only consecutive for duration task | |
| # Preprocessing parameters (adaptive noise-floor thresholding) | |
| threshold_strategy: "noise_floor" # Adaptive per-clip (recommended) | |
| noise_floor_percentile: 2.0 # Use 2nd percentile as noise floor | |
| noise_floor_delta_db: 5.0 # Threshold = noise_floor + 5dB | |
| min_sound_duration_ms: 25 # Filter transient spikes | |
| # Gap multipliers | |
| multiplier_longest: 1.5 # Target must be ≥ 1.5x max background | |
| multiplier_shortest: 0.75 # Target must be ≤ 0.75x min background (changed from 0.5) | |
| min_effective_duration_per_source: 1.0 # Minimum duration per source (seconds) | |
| reject_if_gap_not_met: true | |
| sample_different_clips_same_class: true | |
| ``` | |
| #### ORDER Task | |
| ```yaml | |
| order: | |
| enabled: true | |
| task_duration_size: 2.0 | |
| max_clips_per_sample: 10 # Cap for maximum clips to join | |
| question_types: ["first", "last", "second", "second_last", "after", "before"] | |
| min_clips_for_second_questions: 3 # "second" and "second_last" require ≥3 clips | |
| allow_source_repetition: false # Each clip from unique source | |
| # CAPACITY-AWARE QUESTION TYPE BALANCING: | |
| # - Each question type appears equally across samples | |
| # - Advanced types (second, second_last) assigned to high-capacity samples | |
| # - Basic types (first, last, after, before) for lower-capacity samples | |
| # - NO n_clips balancing: randomly samples from [max(2, max_clips-3), max_clips_per_sample] | |
| ``` | |
| #### VOLUME Task | |
| ```yaml | |
| volume: | |
| enabled: true | |
| task_duration_size: 2.0 | |
| max_clips_per_sample: 10 # Cap for maximum clips with different volumes | |
| question_types: ["max_loudness", "min_loudness"] | |
| # Normalization (CRITICAL for controlled volume comparison) | |
| normalize_to_baseline: true | |
| baseline_dBFS: -20.0 # All clips normalized to this level first | |
| use_lufs: false # DISABLED - LUFS makes everything same perceived loudness! | |
| baseline_lufs: -23.0 # EBU R128 standard (not used when use_lufs=false) | |
| # Volume gap constraints (multipliers) | |
| multiplier_max_loudness: 4.0 # Max must be ≥ 4x second-loudest (~12 dB) | |
| multiplier_min_loudness: 0.25 # Min must be ≤ 0.25x second-softest (~12 dB) | |
| reject_if_gap_not_met: true | |
| # Source clip options | |
| use_same_clip_different_volumes: false # Use different clips (not same clip repeated) | |
| repetitions_per_source: [2, 3, 4] # If same clip used, how many repetitions | |
| # QUESTION TYPE BALANCING: Each question type appears equally across samples | |
| # NO n_clips balancing: randomly samples from [max(2, max_clips-3), max_clips_per_sample] | |
| ``` | |
| --- | |
| ## ESC-50 Preprocessing (Duration Task Only) | |
| **File**: `preprocess_esc50.py` | |
| **Purpose**: Preprocess ESC-50 clips for duration task by detecting actual sound regions and trimming silence. | |
| ### Why Preprocessing? | |
| The DURATION task compares sound durations. Raw ESC-50 clips have variable amounts of leading/trailing silence, which would make duration comparisons ambiguous. Preprocessing: | |
| 1. **Detects actual sound regions** using adaptive amplitude thresholding | |
| 2. **Trims leading and trailing silence** (preserves internal structure) | |
| 3. **Calculates effective duration** (sum of all sound regions) | |
| 4. **Generates metadata CSV** with per-clip durations | |
| ### Preprocessing Pipeline | |
| ``` | |
| Raw ESC-50 clip (5s with silence) | |
| ↓ | |
| 1. Load audio and convert to amplitude array | |
| 2. Compute RMS envelope (frame-by-frame energy) | |
| 3. Convert RMS to dB values | |
| 4. Apply adaptive threshold strategy | |
| 5. Detect contiguous sound regions | |
| 6. Trim edges (only if silence >= 100ms) | |
| 7. Calculate effective duration | |
| 8. Save trimmed audio + metadata | |
| ``` | |
| ### Adaptive Noise-Floor Thresholding | |
| The preprocessing uses an **adaptive per-clip threshold** strategy: | |
| ```python | |
| # Strategy: 'noise_floor' (adaptive, recommended) | |
| noise_floor_db = np.percentile(db_values, noise_floor_percentile) # e.g., 2nd percentile | |
| absolute_threshold = noise_floor_db + noise_floor_delta_db # e.g., +5 dB above noise floor | |
| ``` | |
| **Key Parameters** (from `config.yaml`): | |
| ```yaml | |
| duration: | |
| threshold_strategy: "noise_floor" # Adaptive per-clip (recommended) | |
| noise_floor_percentile: 2.0 # Use 2nd percentile as noise floor estimate | |
| noise_floor_delta_db: 5.0 # Threshold = noise_floor + 5 dB | |
| min_sound_duration_ms: 25 # Filter out transient spikes < 25ms | |
| ``` | |
| **Why Adaptive?** | |
| - Each clip has different background noise levels | |
| - Fixed threshold (e.g., -40 dB) works poorly across diverse sounds | |
| - Adaptive threshold adjusts per-clip based on its own noise floor | |
| **Alternative** (legacy): | |
| ```yaml | |
| threshold_strategy: "peak_relative" # threshold = peak_dB - 20 dB (fixed offset) | |
| amplitude_threshold_db: -20.0 | |
| ``` | |
| ### Edge Trimming Strategy | |
| **ADAPTIVE EDGE-ONLY TRIMMING** - preserves natural periodicity: | |
| ```python | |
| def extract_sound_with_edges_trimmed(audio, regions, min_silence_to_trim_ms=100, buffer_ratio=0.1): | |
| """ | |
| Trim ONLY leftmost and rightmost silence IF significant. | |
| Preserves ALL internal structure (perfect for periodic sounds). | |
| """ | |
| leading_silence_ms = regions[0][0] # Time before first sound | |
| trailing_silence_ms = len(audio) - regions[-1][1] # Time after last sound | |
| # Only trim if silence >= 100ms | |
| if leading_silence_ms >= min_silence_to_trim_ms: | |
| buffer_ms = max(200, int(leading_silence_ms * 0.1)) # Keep 10% as buffer | |
| trim_start_ms = max(0, regions[0][0] - buffer_ms) | |
| else: | |
| trim_start_ms = 0 # Keep from start | |
| # Similar for trailing silence | |
| ... | |
| return audio[trim_start_ms:trim_end_ms] | |
| ``` | |
| **Why Edge-Only?** | |
| - Clock ticks, footsteps, typing have periodic silence between sounds | |
| - Removing internal silences destroys natural rhythm | |
| - Edge trimming removes irrelevant silence while preserving periodicity | |
| ### Output Files | |
| ``` | |
| ESC-50_preprocessed/ | |
| ├── effective_durations.csv | |
| │ ├── filename | |
| │ ├── category | |
| │ ├── raw_duration_s (original 5.0s) | |
| │ ├── final_duration_s (after edge trimming) | |
| │ ├── effective_duration_s (sum of sound regions) | |
| │ ├── num_sound_regions | |
| │ ├── peak_amplitude_db | |
| │ ├── avg_rms_db | |
| │ └── threshold_strategy, noise_floor_percentile, noise_floor_delta_db | |
| └── trimmed_audio/ | |
| ├── 1-100032-A-0.wav (edge-trimmed clips) | |
| └── ... | |
| ``` | |
| ### Running Preprocessing | |
| ```bash | |
| # Using config defaults | |
| python preprocess_esc50.py --config config.yaml | |
| # Override parameters | |
| python preprocess_esc50.py --config config.yaml \ | |
| --threshold-strategy noise_floor \ | |
| --noise-floor-percentile 2.0 \ | |
| --noise-floor-delta-db 5.0 \ | |
| --min-sound-ms 25 | |
| # Don't save trimmed audio (only CSV) | |
| python preprocess_esc50.py --config config.yaml --no-trimmed-audio | |
| ``` | |
| ### Preprocessing Statistics Example | |
| ``` | |
| ESC-50 Preprocessing Summary | |
| ============================================================ | |
| Total clips processed: 2000 | |
| Successfully processed: 2000 | |
| Raw duration statistics: | |
| Mean: 5.000s Std: 0.000s Min: 5.000s Max: 5.000s | |
| Final duration statistics (edges trimmed): | |
| Mean: 4.723s Std: 0.412s Min: 2.134s Max: 5.000s | |
| Effective duration statistics (sum of sound regions): | |
| Mean: 3.856s Std: 0.823s Min: 0.542s Max: 4.982s | |
| Comparison: | |
| Avg effective: 3.856s | |
| Avg final: 4.723s | |
| Difference: 0.867s (internal silences preserved) | |
| Average edge trimming reduction: 5.5% | |
| ``` | |
| ### How Duration Task Uses Preprocessed Data | |
| The `DurationTaskGenerator` loads preprocessed data: | |
| ```python | |
| self.preprocessed_dataset = PreprocessedESC50Dataset( | |
| metadata_csv=config['tasks']['duration']['preprocessed_data_path'] + '/effective_durations.csv', | |
| audio_dir=config['tasks']['duration']['preprocessed_data_path'] + '/trimmed_audio' | |
| ) | |
| # Calculate average effective duration for slot distribution | |
| effective_durations = self.preprocessed_dataset.metadata_df['effective_duration_s'] | |
| self.avg_effective_duration = effective_durations.mean() # ~3.856s | |
| ``` | |
| --- | |
| ## Audio Utilities | |
| Located in `utils/audio_utils.py`. | |
| ### `generate_single_clip_duration(min_duration, max_duration) → float` | |
| **Purpose**: Generate a random target clip duration using UNIFORM sampling. | |
| **Implementation**: | |
| ```python | |
| def generate_single_clip_duration(min_duration: float, max_duration: float) -> float: | |
| return random.uniform(min_duration, max_duration) | |
| ``` | |
| **Mathematical Formulation**: | |
| $$d \sim \text{Uniform}(d_{\min}, d_{\max})$$ | |
| With default values (20s, 60s): | |
| - Mean: $\mu = \frac{20 + 60}{2} = 40$ seconds | |
| - Standard Deviation: $\sigma = \frac{60 - 20}{\sqrt{12}} \approx 11.5$ seconds | |
| --- | |
| ### `get_max_clip_num_to_be_joined(target_duration_s, source_duration_s, min_silence_ms) → Tuple[int, float]` | |
| **Purpose**: Calculate maximum number of source clips that can fit in target duration. | |
| **Returns**: Tuple of (max_clips, remainder_seconds) | |
| **Implementation** (conceptual): | |
| ```python | |
| def get_max_clip_num_to_be_joined(target_s, source_s, min_silence_ms): | |
| silence_s = min_silence_ms / 1000.0 | |
| # Each clip + silence except last | |
| effective_unit = source_s + silence_s | |
| max_clips = int((target_s + silence_s) / effective_unit) | |
| remainder = target_s - (max_clips * source_s + (max_clips - 1) * silence_s) | |
| return max_clips, remainder | |
| ``` | |
| **Mathematical Formula**: | |
| $$N_{\max} = \left\lfloor \frac{T + g}{S + g} \right\rfloor$$ | |
| Where: | |
| - $T$ = target duration (seconds) | |
| - $S$ = source clip duration (5.0s for ESC-50) | |
| - $g$ = minimum silence gap (seconds) | |
| --- | |
| ### `build_count_task_audio(source_audios, source_categories, target_duration, ...)` | |
| **Purpose**: Build the final audio for COUNT task. | |
| **Parameters**: | |
| - `source_audios`: List of AudioSegment objects (one per category) | |
| - `source_categories`: List of category names | |
| - `target_duration`: Target total duration in seconds | |
| - `ordering_mode`: "random" or "consecutive" | |
| - `source_clip_duration_seconds`: Duration of each source clip | |
| - `min_silence_ms`, `max_extra_silence_per_gap_ms`: Silence parameters | |
| **Returns**: Tuple of (final_audio, clip_sequence, build_metadata) | |
| --- | |
| ### `build_duration_task_audio(...)` | |
| **Purpose**: Build audio for DURATION task with slot distribution. | |
| --- | |
| ### `build_clip_sequence_with_silences(clips, target_duration_s, min_silence_ms, max_extra_silence_per_gap_ms, crossfade_ms)` | |
| **Purpose**: Concatenate clips with random silence gaps and smooth crossfades. | |
| **Algorithm**: | |
| 1. Calculate total audio content duration | |
| 2. Calculate minimum required silence: `(n_clips - 1) × min_silence_ms` | |
| 3. Calculate available extra time: `target_duration - total_audio - min_silence` | |
| 4. Distribute extra time randomly across gaps (up to `max_extra_silence_per_gap_ms` per gap) | |
| 5. Build sequence with crossfades: | |
| - Audio → Silence: crossfade for smooth transition | |
| - Silence → Audio: No crossfade (preserves audio start) | |
| **Crossfade Benefits**: | |
| - Smooth transitions between audio and silence | |
| - Reduces clicks/pops at audio boundaries | |
| - Preserves natural sound attack (no crossfade at audio start) | |
| --- | |
| ## Task: COUNT | |
| **File**: `tasks/task_count.py` | |
| **Class**: `CountTaskGenerator` | |
| ### Complete Flow | |
| ``` | |
| CountTaskGenerator.__init__(config, logger) | |
| ↓ | |
| Initialize: | |
| - ESC50Dataset (loads metadata, tracks category usage) | |
| - AudioProcessor | |
| - QuestionGenerator | |
| - LLMQuestionGenerator (if enabled) | |
| ↓ | |
| generate_dataset() | |
| ↓ | |
| 1. num_samples = calculate_num_samples_for_task(task_duration_hours, min, max) | |
| 2. Create balanced_answers list from num_clips_per_sample | |
| 3. Shuffle balanced_answers | |
| 4. For each sample: | |
| generate_sample(sample_id, target_unique_count=balanced_answers[i]) | |
| 5. Save CSVs | |
| ``` | |
| ### Key Method: `generate_sample(sample_id, target_unique_count)` | |
| **Pipeline**: | |
| 1. Generate random target duration: `clip_duration_seconds = generate_single_clip_duration(min, max)` | |
| 2. Calculate max clips: `max_clips, remainder = get_max_clip_num_to_be_joined(...)` | |
| 3. Cap `n_unique_audios` at min(target_unique_count, max_clips, 50) | |
| 4. Select categories: `selected_categories = dataset.get_least_used_categories(n_unique_audios)` | |
| 5. Track usage: Increment `category_usage_counts` for each selected category | |
| 6. Sample one file per category: `dataset.sample_file_from_category(category)` | |
| 7. Load source audios | |
| 8. Build final audio: `build_count_task_audio(source_audios, categories, target_duration, ordering_mode, ...)` | |
| 9. Export audio file | |
| 10. Generate MCQ and open-text questions | |
| 11. Return metadata dict | |
| ### Balanced Answer Distribution (Updated with max_clips_per_sample) | |
| ```python | |
| # In generate_dataset() | |
| max_clips_per_sample = self.task_config.get('max_clips_per_sample', 10) # Single number: 10 | |
| possible_answers = list(range(1, max_clips_per_sample + 1)) # [1, 2, 3, ..., 10] | |
| samples_per_answer = num_samples // len(possible_answers) | |
| remainder = num_samples % len(possible_answers) | |
| balanced_answers = [] | |
| for answer in possible_answers: | |
| count = samples_per_answer + (1 if remainder > 0 else 0) | |
| balanced_answers.extend([answer] * count) | |
| remainder = max(0, remainder - 1) | |
| random.shuffle(balanced_answers) | |
| ``` | |
| **For 90 samples, max_clips_per_sample=10**: Each answer (1-10) appears exactly 9 times. | |
| ### Silence Reduction Strategy (NEW) | |
| Each sample's target answer is capped at what actually fits in the duration: | |
| ```python | |
| # In generate_sample() | |
| max_clips, _ = get_max_clip_num_to_be_joined(clip_duration_seconds, source_clip_duration, min_silence_ms) | |
| if target_unique_count is not None: | |
| # Cap target at what actually fits (reduces silence) | |
| n_unique_audios = min(target_unique_count, max_clips, len(CATEGORIES)) | |
| ``` | |
| **Example**: | |
| - Target answer from balanced pool: **8 unique sounds** | |
| - Duration allows: **max_clips = 7** | |
| - Actual n_unique_audios: **min(8, 7) = 7** ✓ (uses max possible, reduces silence) | |
| **Why?** Prevents excessive silence when target exceeds what fits in duration. | |
| --- | |
| ## Task: DURATION | |
| **File**: `tasks/task_duration.py` | |
| **Class**: `DurationTaskGenerator` | |
| ### Complete Flow | |
| ``` | |
| DurationTaskGenerator.__init__(config, logger) | |
| ↓ | |
| Initialize: | |
| - PreprocessedESC50Dataset (uses effective_durations.csv) | |
| - Calculate avg_effective_duration from preprocessed data | |
| - AudioProcessor, QuestionGenerator | |
| - Load multiplier_longest, multiplier_shortest from config | |
| ↓ | |
| generate_dataset() | |
| ↓ | |
| 1. num_samples = calculate_num_samples_for_task(...) | |
| 2. Create balanced question types: ["longest"] * 45 + ["shortest"] * 45 | |
| 3. Shuffle balanced_types | |
| 4. While len(samples) < num_samples: | |
| generate_sample(sample_idx, question_type=balanced_types[idx]) | |
| If returns None → increment rejection_count, continue | |
| 5. Save CSVs | |
| ``` | |
| ### Key Methods | |
| #### `_calculate_max_clips_and_sources(target_duration_s, question_type)` | |
| **Purpose**: Determine valid number of sources based on question type and duration. | |
| **For LONGEST**: | |
| - Target needs ≥2 clips to beat backgrounds by 1.5x | |
| - `min_valid_sources = 2` | |
| - `max_valid_sources = max_clips - 2 + 1` | |
| **For SHORTEST**: | |
| - Target gets 1 clip | |
| - Each background needs ≥2 clips to be 2x target | |
| - `max_valid_sources = 1 + (max_clips - 1) // 2` | |
| ```python | |
| # Filter config values to valid range, then pick RANDOMLY | |
| valid_config_sources = [n for n in num_sources_config if min_valid <= n <= max_valid] | |
| n_sources = random.choice(valid_config_sources) | |
| ``` | |
| #### `_try_generate_sample(sample_id, question_type)` | |
| **Full Algorithm**: | |
| 1. Generate target duration: `generate_single_clip_duration(min, max)` | |
| 2. Calculate max_clips and n_sources: `_calculate_max_clips_and_sources(...)` | |
| 3. Select target category (least used) | |
| 4. Select background categories (from remaining least used) | |
| 5. Calculate slot distribution based on question_type | |
| 6. For each category, select source files and generate clip durations | |
| 7. Load and trim clips | |
| 8. Calculate total effective duration per category | |
| 9. Verify gap constraint | |
| 10. If gap not satisfied, try `_try_improve_slot_distribution()` | |
| 11. If still not satisfied, return None (triggers retry) | |
| 12. Build audio and generate questions | |
| 13. Return metadata | |
| #### `_try_improve_slot_distribution(slot_distribution, durations, question_type, max_clips)` | |
| **Purpose**: Redistribute slots to satisfy gap constraint. | |
| --- | |
| ## Task: ORDER | |
| **File**: `tasks/task_order.py` | |
| **Class**: `OrderTaskGenerator` | |
| ### Complete Flow | |
| ``` | |
| OrderTaskGenerator.__init__(config, logger) | |
| ↓ | |
| Initialize ESC50Dataset, AudioProcessor, QuestionGenerator | |
| ↓ | |
| generate_dataset() | |
| ↓ | |
| 1. Generate sample durations upfront (exact fill) | |
| 2. num_samples = len(sample_durations) | |
| 3. Create balanced question_types distribution | |
| 4. For each sample: | |
| generate_sample(sample_id, target_question_type=balanced_types[i]) | |
| → n_clips randomly selected from [max(2, max_clips-3), min(max_clips, max_clips_per_sample)] | |
| 5. Save CSVs | |
| ``` | |
| ### Key Method: `_get_valid_question_types(n_clips)` | |
| Filters question types based on clip count: | |
| - `second`, `second_last`: require `n_clips >= min_clips_for_second_questions` (default: 4) | |
| - `after`, `before`: require `n_clips >= 2` | |
| - `first`, `last`: always valid | |
| ### Key Method: `generate_sample(sample_id, target_question_type, target_duration_seconds)` | |
| **Algorithm**: | |
| 1. Use pre-generated `target_duration_seconds` (from sample_durations) | |
| 2. Calculate max_clips from duration: `get_max_clip_num_to_be_joined(...)` | |
| 3. **Silence reduction - randomly select n_clips**: | |
| ```python | |
| min_clips = max(2, max_clips - 3) | |
| max_clips_allowed = min(max_clips, max_clips_per_sample, len(CATEGORIES)) | |
| if min_clips > max_clips_allowed: # Handle edge case | |
| min_clips = max_clips_allowed | |
| n_clips = random.randint(min_clips, max_clips_allowed) | |
| ``` | |
| 4. Get valid question types for n_clips | |
| 5. Select answer position based on question type: | |
| - `first` → position 0 | |
| - `last` → position n_clips - 1 | |
| - `second` → position 1 | |
| - `second_last` → position n_clips - 2 | |
| - `after` → random position 1 to n-1 | |
| - `before` → random position 0 to n-2 | |
| 6. Select categories using least-used balancing (answer first, then others) | |
| 7. Build audio with `build_clip_sequence_with_silences` (includes crossfade) | |
| 8. Generate questions including sequence question | |
| 9. Return metadata | |
| **Silence Reduction**: Target n_clips is capped at `max_clips` to avoid excessive silence. | |
| --- | |
| ## Task: VOLUME | |
| **File**: `tasks/task_volume.py` | |
| **Class**: `VolumeTaskGenerator` | |
| ### Complete Flow | |
| ``` | |
| VolumeTaskGenerator.__init__(config, logger) | |
| ↓ | |
| Initialize ESC50Dataset, AudioProcessor, QuestionGenerator | |
| Load multiplier_max_loudness, multiplier_min_loudness, baseline normalization settings | |
| ↓ | |
| generate_dataset() | |
| ↓ | |
| 1. Generate sample durations upfront (exact fill) | |
| 2. num_samples = len(sample_durations) | |
| 3. Create balanced clips_count_pool from 2 to max_clips_per_sample | |
| 4. Create balanced question_types: ["max_loudness"] * N/2 + ["min_loudness"] * N/2 | |
| 5. Shuffle both pools | |
| 6. Store clips_count_pool as instance variable | |
| 7. For each sample: | |
| generate_sample(sample_id, target_question_type=balanced_types[i]) | |
| → Uses clips_count_pool.pop(0) internally, capped at max_clips_that_fit | |
| → Normalizes clips to baseline, applies volume adjustments | |
| → Verifies gap constraints (up to 10 attempts) | |
| 8. Save CSVs | |
| ``` | |
| ### Key Methods | |
| #### `_normalize_to_baseline(audio)` | |
| ```python | |
| def _normalize_to_baseline(self, audio): | |
| if not self.normalize_to_baseline: | |
| return audio | |
| change_in_dBFS = self.baseline_dBFS - audio.dBFS | |
| return audio.apply_gain(change_in_dBFS) | |
| ``` | |
| #### `_verify_loudness_gap(volume_levels, question_type)` | |
| **For MAX_LOUDNESS**: | |
| ```python | |
| required_gap_dB = 20 * math.log10(self.multiplier_max_loudness) # ≈ 3.52 dB | |
| actual_gap_dB = max_level - second_max | |
| gap_satisfied = actual_gap_dB >= required_gap_dB | |
| ``` | |
| **For MIN_LOUDNESS**: | |
| ```python | |
| required_gap_dB = abs(20 * math.log10(self.multiplier_min_loudness)) # ≈ 6.02 dB | |
| actual_gap_dB = second_min - min_level | |
| gap_satisfied = actual_gap_dB >= required_gap_dB | |
| ``` | |
| #### Volume Level Generation | |
| Volume levels are generated to satisfy gap constraints: | |
| - For `max_loudness`: target gets +gap_dB above baseline, backgrounds at/below baseline | |
| - For `min_loudness`: target gets -gap_dB below baseline, backgrounds at/above baseline | |
| --- | |
| ## Deterministic Balancing Mechanisms | |
| ### Overview | |
| The pipeline ensures balanced distributions across multiple dimensions with **capacity-aware assignment**. | |
| ### 1. Capacity-Aware Answer Balancing (COUNT Task) | |
| Each possible answer (1-10) appears equally often, but **higher targets are assigned to samples with higher capacity**. | |
| ```python | |
| # Calculate capacity for each sample | |
| for duration in sample_durations: | |
| max_clips, _ = get_max_clip_num_to_be_joined(duration, source_clip_duration, min_silence_ms) | |
| max_for_sample = min(max_clips, max_clips_per_sample, len(CATEGORIES)) | |
| sample_max_clips.append(max_for_sample) | |
| # Create balanced pool | |
| possible_answers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | |
| samples_per_answer = num_samples // len(possible_answers) | |
| remainder = num_samples % len(possible_answers) | |
| assignment_pool = [] | |
| for answer in possible_answers: | |
| count = samples_per_answer + (1 if remainder > 0 else 0) | |
| assignment_pool.extend([answer] * count) | |
| remainder = max(0, remainder - 1) | |
| # Sort samples by capacity (descending) | |
| sample_info.sort(key=lambda x: x[2], reverse=True) | |
| # Sort pool descending - assign high targets first | |
| assignment_pool.sort(reverse=True) | |
| # Assign targets, clamped to capacity | |
| for idx, (sample_idx, duration, capacity) in enumerate(sample_info): | |
| target = min(assignment_pool[idx], capacity) | |
| balanced_assignments[sample_idx] = target | |
| ``` | |
| **Guarantee**: Each answer value appears equally, and high targets go to samples that can fit them. | |
| ### 2. Capacity-Aware Question Type Balancing (ORDER Task) | |
| ORDER task uses **capacity-aware balancing** - advanced question types assigned to high-capacity samples. | |
| ```python | |
| # Separate question types by requirements | |
| basic_types = ['first', 'last', 'after', 'before'] # Need >= 2 clips | |
| advanced_types = ['second', 'second_last'] # Need >= min_clips_for_second (e.g., 3) | |
| # Sort samples by capacity (descending) | |
| sample_info.sort(key=lambda x: x[2], reverse=True) | |
| # Build assignment pool - advanced types first | |
| samples_per_type = num_samples // len(question_types) | |
| remainder = num_samples % len(question_types) | |
| assignment_pool = [] | |
| # Add advanced types first (for high-capacity samples) | |
| for qtype in advanced_types: | |
| count = samples_per_type + (1 if remainder > 0 else 0) | |
| assignment_pool.extend([qtype] * count) | |
| remainder = max(0, remainder - 1) | |
| # Then basic types | |
| for qtype in basic_types: | |
| count = samples_per_type + (1 if remainder > 0 else 0) | |
| assignment_pool.extend([qtype] * count) | |
| remainder = max(0, remainder - 1) | |
| # Assign with validation | |
| for idx, (sample_idx, duration, capacity) in enumerate(sample_info): | |
| target_qtype = assignment_pool[idx] | |
| valid_types = _get_valid_question_types(capacity) | |
| if target_qtype not in valid_types: | |
| # Downgrade to valid type | |
| target_qtype = random.choice(valid_types) | |
| balanced_assignments[sample_idx] = target_qtype | |
| ``` | |
| ### 3. Simple Question Type Balancing (DURATION, VOLUME Tasks) | |
| ```python | |
| # DURATION: 2 types → N/2 each | |
| # VOLUME: 2 types → N/2 each | |
| samples_per_type = num_samples // len(question_types) | |
| remainder = num_samples % len(question_types) | |
| balanced_types = [] | |
| for qtype in question_types: | |
| count = samples_per_type + (1 if remainder > 0 else 0) | |
| balanced_types.extend([qtype] * count) | |
| remainder = max(0, remainder - 1) | |
| random.shuffle(balanced_types) | |
| ``` | |
| ### 4. Category Usage Balancing | |
| All 50 ESC-50 categories are used equally via least-used selection: | |
| ```python | |
| def get_least_used_categories(self, n: int, exclude: List[str] = None) -> List[str]: | |
| # Sort categories by usage count | |
| sorted_cats = sorted( | |
| self.category_usage_counts.items(), | |
| key=lambda x: (x[1], x[0]) # Sort by count, then alphabetically for ties | |
| ) | |
| # Filter excluded and return first n | |
| available = [cat for cat, _ in sorted_cats if cat not in (exclude or [])] | |
| return available[:n] | |
| ``` | |
| Each task calls `reset_category_usage()` at the start to ensure independent balancing. | |
| ### 5. N_Clips Selection Strategy | |
| **COUNT Task**: Uses capacity-aware answer balancing (see #1 above) | |
| **ORDER and VOLUME Tasks**: Use **silence reduction strategy** (NOT balanced): | |
| ```python | |
| # Randomly sample n_clips from valid range to minimize silence | |
| min_clips = max(2, max_clips - 3) | |
| max_clips_allowed = min(max_clips, max_clips_per_sample, len(CATEGORIES)) | |
| if min_clips > max_clips_allowed: | |
| min_clips = max_clips_allowed # Handle edge case | |
| n_clips = random.randint(min_clips, max_clips_allowed) | |
| ``` | |
| This maximizes clip usage within the allowed range, minimizing excessive silence. | |
| --- | |
| ## Rejection Logic and Retry Mechanisms | |
| ### When Samples Are Rejected | |
| Rejections occur only in tasks with gap constraints: | |
| 1. **DURATION Task**: Gap constraint not satisfied | |
| - LONGEST: target_duration < max_background × 1.5 | |
| - SHORTEST: target_duration > min_background × 0.5 | |
| 2. **VOLUME Task**: Gap constraint not satisfied | |
| - MAX_LOUDNESS: actual_gap_dB < required_gap_dB (3.52 dB) | |
| - MIN_LOUDNESS: actual_gap_dB < required_gap_dB (6.02 dB) | |
| ### DURATION Task Retry Logic | |
| ```python | |
| def generate_dataset(self): | |
| all_metadata = [] | |
| sample_idx = 0 | |
| type_idx = 0 | |
| while len(all_metadata) < num_samples and type_idx < len(balanced_types) * 2: | |
| question_type = balanced_types[type_idx % len(balanced_types)] | |
| metadata = self.generate_sample(sample_idx, question_type) | |
| if metadata is not None: | |
| all_metadata.append(metadata) | |
| sample_idx += 1 | |
| # If None, sample was rejected - just move to next | |
| type_idx += 1 | |
| ``` | |
| ### Rejection Rate Calculation | |
| $$\text{Rejection Rate} = \frac{\text{rejections}}{\text{rejections} + \text{successes}} \times 100\%$$ | |
| --- | |
| ## Complete Task Creation Explanation | |
| ### How Each Task Is Generated (Step-by-Step) | |
| #### COUNT TASK - "How many unique sounds?" | |
| **Goal**: Create audio with N unique sound sources, ask how many distinct sounds exist. | |
| **Process**: | |
| 1. **Preprocessing**: None (uses raw ESC-50 clips) | |
| 2. **Duration Generation**: `target_duration ~ Uniform(20s, 60s)` per sample | |
| 3. **Calculate Max Clips**: `max_clips = get_max_clip_num_to_be_joined(target_duration, 5s, 100ms)` | |
| - Example: 45s duration → ~8 clips of 5s each with 100ms silence between | |
| 4. **Balanced Answer Selection**: Pre-generated pool of answers [1,2,3,...,10] balanced equally | |
| - Target answer (e.g., 5 unique sounds) selected from pool | |
| 5. **Silence Reduction**: Cap target at `min(target_answer, max_clips)` | |
| - If target=8 but max_clips=6 → use 6 (prevents excessive silence) | |
| 6. **Category Selection**: Pick N least-used categories from ESC-50 (balancing) | |
| 7. **Audio Construction**: | |
| - Load one file per category | |
| - Calculate repetitions needed: `total_clips = max_clips` | |
| - Distribute repetitions across N sources | |
| - **Ordering mode**: | |
| - `random`: Shuffle clips (A B A C B...) - harder, tests recognition | |
| - `consecutive`: Group same-source (AAA BBB CCC) - easier | |
| 8. **Silence Insertion**: | |
| - Minimum 100ms silence between EVERY clip | |
| - Extra silence (up to 500ms per gap) distributed from remainder | |
| - **Crossfade**: 50ms within same-source, 500ms at audio-silence boundaries | |
| 9. **Question Generation**: MCQ + open-text asking "How many unique sounds?" | |
| 10. **Export**: Save audio WAV + metadata | |
| **Example**: | |
| - Target duration: 40s | |
| - Max clips that fit: 7 clips (7×5s + 6×0.1s = 35.6s) | |
| - Target answer: 3 unique sounds | |
| - Actual: 3 unique sounds (7 total clips: 3+2+2 repetitions) | |
| - Ordering: Random shuffle → [A B A C B A C] | |
| - Result: Audio with 3 distinct sounds, some repeated, with silences and crossfades | |
| #### DURATION TASK - "Which sound is longest/shortest?" | |
| **Goal**: Create audio where one sound has clearly longest/shortest duration compared to others. | |
| **Process**: | |
| 1. **Preprocessing** (preprocess_esc50.py - REQUIRED): | |
| - Load raw ESC-50 clips | |
| - Detect sound regions using adaptive noise-floor thresholding | |
| - Trim leading/trailing silence (preserve internal structure) | |
| - Calculate effective duration per clip | |
| - Save trimmed audio + effective_durations.csv | |
| 2. **Duration Generation**: `target_duration ~ Uniform(20s, 60s)` per sample | |
| 3. **Calculate Max Clips**: Based on average effective duration (~3.86s) | |
| 4. **Determine N Sources**: Based on question type and max_clips | |
| - **LONGEST**: Target needs ≥2 clips, backgrounds get 1 each → `n_sources ≤ max_clips - 1` | |
| - **SHORTEST**: Target gets 1 clip, backgrounds need ≥2 each → `n_sources ≤ 1 + (max_clips-1)//2` | |
| 5. **Category Selection**: Pick target + backgrounds from least-used categories | |
| 6. **Slot Distribution**: Allocate clips to each source | |
| - LONGEST: Give most clips to target, 1 to each background | |
| - SHORTEST: Give 1 to target, multiple to each background | |
| 7. **Clip Selection**: For each source, select clips from preprocessed dataset | |
| 8. **Gap Verification**: | |
| - LONGEST: `target_duration ≥ max_background × 1.5` ✓ | |
| - SHORTEST: `target_duration ≤ min_background × 0.75` ✓ | |
| - If gap not satisfied: Try redistributing slots, or reject sample | |
| 9. **Audio Construction**: | |
| - Load trimmed clips | |
| - Concatenate with consecutive ordering (preserve periodicity) | |
| - Insert silences with crossfades | |
| 10. **Question Generation**: "Which sound is longest/shortest?" | |
| 11. **Export**: Audio + metadata | |
| **Example**: | |
| - Question type: LONGEST | |
| - Target duration: 50s, max_clips: 12 | |
| - N sources: 4 (target + 3 backgrounds) | |
| - Slot distribution: Target=6 clips (6×3.8s=22.8s), Backgrounds=2 clips each (2×3.8s=7.6s) | |
| - Gap check: 22.8s ≥ 7.6s × 1.5 = 11.4s ✓ | |
| - Result: Target sound clearly longest | |
| #### ORDER TASK - "Which sound is first/last/after X?" | |
| **Goal**: Create ordered sequence of sounds, ask about temporal relationships. | |
| **Process**: | |
| 1. **Preprocessing**: None (uses raw ESC-50) | |
| 2. **Duration Generation**: Pre-generated durations to exactly fill task duration | |
| 3. **Calculate Max Clips**: `get_max_clip_num_to_be_joined(target_duration, 5s, 100ms)` | |
| 4. **Balanced N_Clips Selection**: Pre-generated pool [2,3,4,...,10] balanced equally | |
| - Target n_clips (e.g., 5) selected from pool | |
| - Capped at `min(target_n_clips, max_clips)` (silence reduction) | |
| 5. **Question Type Selection**: From balanced pool (first, last, second, after, before, second_last) | |
| 6. **Answer Position Determination**: Based on question type | |
| - `first` → position 0 | |
| - `last` → position n_clips-1 | |
| - `second` → position 1 | |
| - `second_last` → position n_clips-2 | |
| - `after`/`before` → random valid position | |
| 7. **Category Selection**: Answer category at determined position, others from least-used | |
| 8. **Audio Construction**: | |
| - Load one clip per position | |
| - Build sequence with silences (min 100ms + random extra up to 500ms per gap) | |
| - **Crossfade**: 500ms at audio-silence boundaries for smooth transitions | |
| 9. **Question Generation**: | |
| - MCQ: "Which sound is first?" with 4 options | |
| - Open-text: "What is the first sound?" + full sequence | |
| 10. **Export**: Audio + metadata | |
| **Example**: | |
| - Target n_clips: 4, max_clips: 8 → use 4 ✓ | |
| - Question: "Which sound is second?" | |
| - Answer position: 1 (0-indexed) | |
| - Sequence: [dog, cat, bird, rain] → Answer: cat | |
| - Audio: 4 clips in order with silences and crossfades | |
| #### VOLUME TASK - "Which sound is loudest/softest?" | |
| **Goal**: Create audio with clips at different volume levels, ask about loudness comparison. | |
| **Process**: | |
| 1. **Preprocessing**: None (uses raw ESC-50) | |
| 2. **Duration Generation**: Pre-generated durations | |
| 3. **Calculate Max Clips**: `get_max_clip_num_to_be_joined(...)` | |
| 4. **Balanced N_Clips Selection**: From pool [2,3,...,10], capped at max_clips | |
| 5. **Question Type Selection**: "max_loudness" or "min_loudness" (balanced 50/50) | |
| 6. **Volume Level Generation**: Create n_clips volume adjustments (in dB) | |
| - Ensure gap constraint (multiplier 4.0 for max, 0.25 for min) | |
| - Example: [+12dB, 0dB, -6dB] → max at +12dB has ≥12dB gap from second | |
| 7. **Gap Verification** (up to 10 attempts): | |
| - MAX: `max_level - second_max ≥ 20×log10(4.0) ≈ 12dB` | |
| - MIN: `second_min - min_level ≥ 20×log10(4.0) ≈ 12dB` | |
| - If not satisfied: Regenerate levels or reject | |
| 8. **Category Selection**: Answer at determined position, others from least-used | |
| 9. **Audio Construction**: | |
| - Load clips | |
| - **CRITICAL: Normalize all to baseline (-20 dBFS)** → ensures controlled comparison | |
| - Apply volume adjustments to normalized clips | |
| - Concatenate with silences and crossfades | |
| 10. **Question Generation**: "Which sound has maximum/minimum loudness?" | |
| 11. **Export**: Audio + metadata with volume levels | |
| **Example**: | |
| - Target n_clips: 3, max_clips: 6 → use 3 ✓ | |
| - Question: "max_loudness" | |
| - Volume levels: [+12dB, 0dB, -6dB] | |
| - Gap check: 12 - 0 = 12dB ≥ 12dB ✓ | |
| - Process: Normalize all clips to -20dBFS, then adjust to [-8dBFS, -20dBFS, -26dBFS] | |
| - Result: First sound clearly loudest | |
| ### Key Innovations | |
| 1. **Crossfade Everywhere**: Smooth transitions at audio-silence boundaries (500ms), small crossfade within same-source repetitions (50ms) | |
| 2. **Adaptive Preprocessing**: Noise-floor thresholding adapts per-clip (duration task) | |
| 3. **Silence Reduction**: ORDER/VOLUME tasks sample n_clips from [max_clips-3, max_clips_per_sample] to minimize silence | |
| 4. **Balanced Distribution**: | |
| - **COUNT**: Balances answers (1 to max_clips_per_sample) + question types | |
| - **ORDER/VOLUME**: Balances question types only (n_clips uses silence reduction) | |
| 5. **Category Balancing**: Least-used selection ensures all 50 ESC-50 categories used evenly | |
| 6. **Gap Constraints**: Mathematical guarantees for duration/volume comparisons | |
| 7. **Exact Duration Filling**: Pre-generate sample durations to exactly fill task duration (no wasted time) | |
| --- | |
| ## Command-Line Arguments | |
| ### Main Pipeline (`main.py`) | |
| ```bash | |
| python main.py [OPTIONS] | |
| Options: | |
| --config, -c PATH Path to config YAML (default: config.yaml) | |
| --tasks, -t TASKS Specific tasks to run (choices: count, duration, order, volume) | |
| --output, -o PATH Custom output directory (overrides config) | |
| Examples: | |
| # Run all enabled tasks with default config | |
| python main.py | |
| # Run specific tasks only | |
| python main.py --tasks count order | |
| # Use custom config and output | |
| python main.py --config my_config.yaml --output ./my_dataset | |
| ``` | |
| ### Preprocessing Script (`preprocess_esc50.py`) | |
| ```bash | |
| python preprocess_esc50.py [OPTIONS] | |
| Options: | |
| --config PATH Path to config YAML (default: config.yaml) | |
| --threshold-strategy STRATEGY "noise_floor" or "peak_relative" | |
| --threshold-db FLOAT Threshold in dB (for peak_relative) | |
| --noise-floor-percentile FLOAT Percentile for noise floor estimation | |
| --noise-floor-delta-db FLOAT Delta above noise floor in dB | |
| --min-sound-ms INT Minimum sound duration in ms | |
| --no-trimmed-audio Skip saving trimmed audio files | |
| --output-dir PATH Custom output directory | |
| Examples: | |
| # Use config defaults | |
| python preprocess_esc50.py --config config.yaml | |
| # Override threshold parameters | |
| python preprocess_esc50.py --config config.yaml \ | |
| --threshold-strategy noise_floor \ | |
| --noise-floor-percentile 2.0 \ | |
| --noise-floor-delta-db 5.0 \ | |
| --min-sound-ms 25 | |
| # Generate metadata only (no trimmed audio) | |
| python preprocess_esc50.py --config config.yaml --no-trimmed-audio | |
| ``` | |
| --- | |
| ## Summary | |
| The TREA 2.0 pipeline generates balanced, constraint-satisfying audio QA samples through: | |
| 1. **Preprocessing** (Duration only): Adaptive noise-floor thresholding + edge trimming | |
| 2. **Exact Duration Filling**: Pre-generate sample durations to sum exactly to task duration | |
| 3. **Capacity-Aware Balancing**: | |
| - **COUNT**: High answer targets → high-capacity samples | |
| - **ORDER**: Advanced question types → high-capacity samples | |
| 4. **Silence Reduction**: ORDER/VOLUME randomly sample n_clips from [max_clips-3, max_clips_per_sample] | |
| 5. **Crossfade Transitions**: Smooth audio-silence boundaries (500ms) + within-source (50ms) | |
| 6. **Category Balancing**: Least-used selection ensures even ESC-50 category distribution | |
| 7. **Gap Constraints**: Mathematical guarantees (1.5x for longest, 0.75x for shortest, 4.0x/0.25x for volume) | |
| 8. **Retry Mechanisms**: Failed samples rejected, pipeline continues until target count reached | |
| All randomness is seeded (`random_seed: 42`) for reproducibility. | |