multihop_tasks_walkthrough.md

Multihop Temporal Reasoning Tasks — Detailed Walkthrough

Architecture Overview

All 8 new task generators follow a shared architecture built on MultihopBaseGenerator, which provides:

• Scene building: Constructing multi-event audio sequences with controlled order, durations, volumes, and silences
• Dataset generation: Sample-loop orchestration with balanced question type distribution
• CSV output: Standardized MCQ, open-text, and metadata CSV writing

graph TD
    A[MultihopBaseGenerator] --> B[ConditionalCountTaskGenerator]
    A --> C[ConditionalDurationTaskGenerator]
    A --> D[BetweenEventsTaskGenerator]
    A --> E[EventDensityTaskGenerator]
    A --> F[DurationGapTaskGenerator]
    A --> G[TemporalArithmeticTaskGenerator]
    A --> H[TemporalLoudnessTaskGenerator]
    A --> I[MultiHopTaskGenerator]
    
    J[ESC50Dataset] --> A
    K[PreprocessedESC50Dataset] --> C
    K --> F
    K --> G
    K --> I
    L[AudioProcessor] --> A
    M[QuestionGenerator] --> A

Key Design Decision: Why a Shared Base?

The existing 7 tasks (count, duration, order, etc.) are standalone classes with no shared base. This worked because each task has fundamentally different audio construction logic (e.g., overlap uses overlay(), during_contains uses temporal containment, silence_gap uses controlled gaps).

The multihop tasks are different — they all build the same kind of audio scene (sequential events with gaps and volume variation) but ask different questions over that scene. The complexity is in the question generation and answer computation, not the audio construction. So a shared base class makes sense here.

---

Base Class: MultihopBaseGenerator

File: multihop_base.py

What it provides

Method	Purpose
__init__	Loads config, initializes dataset/processor/question generator, creates output dirs
build_sequential_scene()	Builds an audio scene with N events, controlled gaps, optional per-event volume
generate_dataset()	Orchestrates sample generation loop with balanced question types
_save_mcq_csv()	Writes MCQ CSV in standard pipeline format
_save_open_text_csv()	Writes open-text CSV in standard pipeline format
_save_metadata_csv()	Writes metadata CSV

Scene Building Logic

┌─────────┐  gap_0  ┌─────────┐  gap_1  ┌─────────┐  gap_2  ┌─────────┐
│ Event 0 │◄──────►│ Event 1 │◄──────►│ Event 2 │◄──────►│ Event 3 │
│  cat_A   │         │  cat_B   │         │  cat_A   │         │  cat_C   │
│  vol: 0dB│         │ vol:-6dB │         │ vol:+3dB │         │ vol:-2dB │
└─────────┘         └─────────┘         └─────────┘         └─────────┘

1. Sample n_events unique categories from ESC-50
2. Load and extend each source clip to fill its time slot via concatenate_to_target_duration()
3. Apply per-event volume adjustments (if specified)
4. Generate controlled gap durations using generate_controlled_gap_durations() (ensures the longest gap is ≥ gap_multiplier × shortest gap)
5. Concatenate: event + silence + event + silence + ...
6. Return per-event metadata (start_ms, end_ms, duration, volume, category)

---

Task 1: ConditionalCountTaskGenerator

File: task_conditional_count.py Family: Multihop Singular Reasoning chain: Temporal condition → Count

What it does

Counts sound events after applying a temporal filter. Unlike the simple COUNT task which asks "how many unique sounds?", this task asks "how many X sounds occur after Y?" — requiring the model to first identify when Y occurs, then count X events only in the filtered region.

Audio Construction

Builds a scene with 4–8 events, intentionally allowing repeated categories (unlike between_events which uses all unique). This is critical because counting repeated occurrences is the core question.

# Example scene: categories = [dog, cat, dog, bird, cat, dog, bird]
# count_after(target=dog, anchor=cat[0]) → 3 (indices 2, 5 after first cat)
# count_before(target=bird, anchor=dog[2]) → 1 (index 3 before third dog)

Question Types & Answer Computation

Type	Logic
count_after	Pick anchor index → count target_sound in categories[anchor+1:]
count_before	Pick anchor index → count target_sound in categories[:anchor]
count_between	Pick two anchor indices → count events in categories[idx1+1:idx2]
count_during	Pick anchor → count all other target_sound occurrences
count_overlap	Pick anchor → count adjacent events (neighbors at ±1 index)

MCQ Generation

Uses QuestionGenerator.generate_count_mcq() — generates 4 integer options including the correct count, with distractors sampled from nearby integers.

---

Task 2: ConditionalDurationTaskGenerator

File: task_conditional_duration.py Family: Multihop Singular Reasoning chain: Temporal condition → Duration comparison

What it does

Compares durations only among events in a temporal subset. "Which sound after Y lasts the longest?" requires: (1) identify Y's position, (2) consider only sounds after Y, (3) compare their durations.

Audio Construction

Uses PreprocessedESC50Dataset (not plain ESC50Dataset) to get effective durations — the actual content duration after silence trimming. This ensures duration comparisons are meaningful (a 5s clip with 3s of trailing silence has effective duration 2s).

# Each event loaded with known effective duration:
fname, fpath, eff_dur = self.dataset.sample_file_from_category_with_duration(cat)

Question Types & Answer Computation

Type	Logic
longest_after	Filter events after anchor → max(effective_durations)
shortest_after	Filter events after anchor → min(effective_durations)
longest_before	Filter events before anchor → max(effective_durations)
shortest_before	Filter events before anchor → min(effective_durations)
repeated_compare	Find category appearing both before AND after anchor → compare their durations

MCQ Generation

Uses QuestionGenerator.generate_category_mcq() — the answer is a sound category name, with distractors drawn from other categories present in the audio.

---

Task 3: BetweenEventsTaskGenerator

File: task_between_events.py Family: Multihop Singular Reasoning chain: Define temporal window → Identify/Count events inside

What it does

Identifies or counts sounds occurring inside the temporal window between two anchor events. "Which sound occurs between X and Y?" requires: (1) find X, (2) find Y, (3) identify events strictly between them.

Audio Construction

Uses all unique categories (no repeats) so that "which sound between X and Y?" has an unambiguous answer. Scene has 5–8 events.

Question Types & Answer Computation

Type	Logic
identify_between	Pick two anchors idx1 < idx2 → randomly select one event from categories[idx1+1:idx2] as answer
count_between	Pick two anchors → len(categories[idx1+1:idx2])
yes_no_between	Pick target → check if target is in the between-window (50/50 true/false balance)

Key Constraint

Anchors are chosen with at least 2 positions apart (idx2 ≥ idx1 + 2) to guarantee at least 1 event between them. Samples with fewer than 3 events are rejected.

---

Task 4: EventDensityTaskGenerator

File: task_event_density.py Family: Multihop Singular Reasoning chain: Segment audio into regions → Compare event counts

What it does

Compares event frequency across temporal regions. "Which half of the audio contains more sound events?" requires counting events in each half and comparing.

Audio Construction

Builds scenes with 6–10 events (larger scenes to make density differences meaningful). Allows repeated categories so label-specific density questions are possible. Categories are shuffled randomly — the natural random distribution creates asymmetry.

Question Types & Answer Computation

Type	Logic
half_density	Split at midpoint → compare len(first_half) vs len(second_half) → answer is "first half" / "second half" / "equal"
before_after_density	Pick anchor → compare count of events before vs after anchor
label_density	Pick anchor + target label → compare count of target_sound before vs after anchor

MCQ Generation

Custom option construction — options are textual ("first half", "second half", "equal", "cannot determine") rather than category names.

---

Task 5: DurationGapTaskGenerator

File: task_duration_gap.py Family: Multihop Inter-Task Reasoning chain: Duration reasoning ↔ Silence-gap reasoning

What it does

Crosses two task families — compares sound event duration against silence gap duration. "Which is longer: the dog sound or the silence after the dog sound?" requires perceiving both the event's length and the gap's length, then comparing.

Audio Construction

Uses PreprocessedESC50Dataset for effective durations. Wider gap range (500–4000ms) compared to other tasks, ensuring gaps are sometimes longer and sometimes shorter than events — making the comparison non-trivial.

Each event's metadata tracks both effective_duration_ms and gap_before_ms / gap_after_ms.

Question Types & Answer Computation

Type	Logic
event_vs_after_gap	Pick event (not last) → compare effective_duration_ms vs gap_after_ms
event_vs_before_gap	Pick event (not first) → compare effective_duration_ms vs gap_before_ms
gap_vs_event	Pick a gap between events A,B → pick third event C → compare gap_duration vs C.effective_duration_ms

---

Task 6: TemporalArithmeticTaskGenerator

File: task_temporal_arithmetic.py Family: Multihop Inter-Task Reasoning chain: Duration + Count/Repetition + Silence reasoning

What it does

Compares aggregated durations — total active time of one label vs another, or total sound time vs total silence time. Requires mentally summing up all occurrences of a repeated sound.

Audio Construction

Uses PreprocessedESC50Dataset. Allows repeated categories (critical for aggregation). Computes two aggregate quantities:

• label_totals: Total effective duration per category (summed across all occurrences)
• total_silence_ms: Sum of all gap durations

Question Types & Answer Computation

Type	Logic
total_label_duration	Pick two categories → compare label_totals[sound1] vs label_totals[sound2]
label_vs_silence	Pick one category → compare label_totals[target] vs total_silence_ms
combined_duration	Pick one category → compare label_totals[target] vs max(gap_durations) (longest single silence)

---

Task 7: TemporalLoudnessTaskGenerator

File: task_temporal_loudness.py Family: Multihop Inter-Task Reasoning chain: Temporal filter → Loudness comparison

What it does

Combines temporal filtering with loudness comparison. "Which sound after Y is the loudest?" requires: (1) identify Y's position, (2) consider only sounds after Y, (3) compare their loudness levels.

Audio Construction

Each event gets a random volume adjustment from volume_range_db (default: [-12, +6] dB). A minimum volume difference (min_volume_diff_db: 3dB) is enforced to ensure the loudest/softest distinction is perceivable.

Actual loudness is measured using LUFS (get_lufs_loudness()) for perceptually accurate loudness comparison.

Question Types & Answer Computation

Type	Logic
loudest_after	Filter events after anchor → max(loudness_lufs)
softest_after	Filter events after anchor → min(loudness_lufs)
loudest_before	Filter events before anchor → max(loudness_lufs)
softest_before	Filter events before anchor → min(loudness_lufs)
repeated_loudness	Find category with multiple occurrences → which occurrence is loudest? Also detects trend (louder/softer over time)

---

Task 8: MultiHopTaskGenerator

File: task_multi_hop.py Family: Multihop Inter-Task Reasoning chain: Explicit 2–3 step chains across different properties

What it does

The most complex task — requires explicit multi-step reasoning across different temporal/acoustic properties. "What sound occurs after the longest sound?" requires: (1) identify the longest sound (duration property), (2) find what comes after it (order property).

Audio Construction

The richest scene — combines ALL controllable dimensions:

• Duration variation via PreprocessedESC50Dataset effective durations
• Volume variation via per-event dB adjustments ([-10, +6] dB range)
• Silence variation via controlled gaps (500–3000ms range, 2.5× multiplier)
• Category repetition for count-based sub-questions

Question Types & Reasoning Chains

Type	Step 1	Step 2	Step 3
after_longest	Find longest duration event	Return next event in sequence	—
before_longest	Find longest duration event	Return previous event	—
after_shortest	Find shortest duration event	Return next event	—
before_loudest	Find loudest event (LUFS)	Return previous event	—
after_longest_gap	Find longest silence gap	Return event after that gap	(or count remaining events)
loudest_after_anchor	Find anchor position	Filter events after anchor	Find loudest among them
longest_before_anchor	Find anchor position	Filter events before anchor	Find longest among them
count_before_loudest	Find loudest event	Filter events before it	Count target_sound occurrences
count_after_longest	Find longest event	Filter events after it	Count target_sound occurrences

Answer Computation Example

# after_longest: "What sound occurs after the longest sound?"
pivot_idx = durations_ms.index(max(durations_ms))  # Step 1: find longest
target_idx = pivot_idx + 1                          # Step 2: get next event
correct = categories[target_idx]                    # Answer: that category

---

Summary: What Makes Each Task Unique

Task	Audio Uniqueness	Question Uniqueness
conditional_count	Repeated categories	Temporal filter → integer count
conditional_duration	PreprocessedESC50 effective durations	Temporal filter → duration superlative
between_events	All unique categories, ≥3 events	Temporal window identification
event_density	6–10 events, asymmetric distribution	Region-based count comparison
duration_gap	Wide gap range (500–4000ms)	Cross-property: event-dur vs silence-dur
temporal_arithmetic	Repeated categories + tracked totals	Aggregated duration arithmetic
temporal_loudness	Per-event volume variation + LUFS measurement	Temporal filter → loudness superlative
multi_hop	All dimensions varied simultaneously	2–3 step reasoning chains