humaneval/README.md

Human Evaluation for the Anticipatory Music Transformer

Generating clips for the qualification round

First we select five clips with melodic content from the Lakh MIDI test set: split f. Selected clips are stored to the qualify directory.

python melody-select.py $DATAPATH/lmd_full/f/ -o qualify -c 5 -s 1 -v

Then we generate accompaniments to these clips. We specify the reference midis (-d option) for the retrieval baseline.

python accompany.py qualify -r -d $DATAPATH/lmd_full/f/

Generating clips for the prompted completion round

We generate prompted completions using an autoregressive model (or an anticipatory autoregressive model) checkpoint stored at $MODELPATH.

First, we randomly select 50 prompts and completions from a collection of completions generated using the FIGARO Music Transformer (stored at $FIGARO). Store these prompts at $PROMPTPATH:

python figaro-select.py $FIGARO -o $PROMPTPATH -c 50 -s 999 -v

Generate completions using a model stored at $MODELPATH and store the results to $PROMPTPATH/$OUTPUT:

python prompt.py $PROMPTPATH $MODELPATH -o $OUTPUT -c 50 -v

Generate completions using an interarrival-time model:

python prompt-interarrival.py $PROMPTPATH $MODELPATH $OUTPUT -c 50 -v

Generating clips for the accompaniment round

We generate accompaniments using an anticipatory autoregressive model checkpoint stored at $MODELPATH.

First, select 50 clips with melodic content:

python melody-select.py $DATAPATH/lmd_full/f/ -o accompany -c 50 -v

Generate anticipatory accompaniments (-a flag):

python accompany.py accompany --model $MODELPATH -av -c 50

Generate the autoregressive baseline (-b flag):

python accompany.py accompany --model $MODELPATH -bv -c 50

Generate the retrieval baseline (-r flag):

python accompany.py accompany -d $DATAPATH/lmd_full/f/ -rv -c 50