README.md

---
license: mit
language:
  - en
tags:
  - dialogue
  - task-oriented-dialogue
  - sentence-embeddings
  - contextual-representations
library_name: pytorch
---

# TRACE — contextual turn encoders (REPRO campaign)

Checkpoints for **TRACE**, a contextual turn encoder that *learns* the update rule
applied to frozen turn embeddings in task-oriented dialogue.

A frozen base encoder $f_1$ maps each utterance to a static vector $e_t$; TRACE
($f_2$) is a Transformer whose **tokens are turns**, and it maps the sequence
$(e_1,\dots,e_T)$ to contextual representations $h_t$. Training is
self-supervised and requires no functional annotation.

All checkpoints come from the **REPRO campaign**: a pre-specified protocol in
which dialogues are partitioned by *near-duplicate cluster*, probes are fit on a
held-out development split, and test labels never inform training, checkpoint or
hyperparameter selection. Every model was retrained from scratch under that
partition.

## Which one to use

| | |
|---|---|
| **Canonical model** | `trace-repro-lite-ar-s42/best` |
| Base encoder (frozen) | `sergioburdisso/dialog2flow-joint-bert-base` |
| Attention mode | autoregressive — the deployable one |
| Parameters | 44.9M (6 layers, 8 heads, 768-dim) |

The AR mode is the one to use in practice: it only attends to the dialogue so
far. Bidirectional checkpoints see the whole window and are included for the
representation-vs-anticipation analysis.

## Naming

```
trace-repro-<recipe>-<mode>-s<seed>[-xlc][-scratch]
             │        │      │      │      └── trained from scratch on the 28M curriculum
             │        │      │      └───────── continued pretraining on ~28M turns
             │        │      └──────────────── training seed: 42, 7, 123
             │        └─────────────────────── ar | bidi
             └──────────────────────────────── lite (6 layers) | deep (12) | gru (recurrent control)
```

Base-ablation checkpoints carry the base in the name instead: `mpnet`, `todbert`.

## Correspondence with the paper

| checkpoints | what they support |
|---|---|
| `trace-repro-{lite,deep}-{ar,bidi}-s{42,7,123}` | main ladder: current- and next-act prediction |
| `trace-repro-gru-ar-s{42,7,123}` | learned recurrent control — isolates attention from recurrence |
| `trace-repro-{mpnet,todbert}-lite-ar-s{42,7,123}` | three-base ablation |
| `trace-repro-lite-ar-s*-xlc`, `-xlc-scratch` | data-scale study (28M-turn curriculum) |
| `trace-repro-deep-ar-s*-xlc` | model + data scale combined |

Untrained controls are not published: they are randomly initialized copies of the
same architectures, reproducible from the configs with seeds 0–4.

## Contents of each folder

```
<checkpoint>/
├── best/               # selected by validation loss — use this one
│   ├── config.json
│   └── model.safetensors
├── config.json         # final epoch
├── model.safetensors
└── trainlog.jsonl      # per-epoch train/val loss
```

## Usage

```python
from contextual_turn_embeddings import ContextualTurnModelV2, encode_dialogues

model = ContextualTurnModelV2.from_pretrained("best", device="cpu").eval()
# embeddings: (n_turns, 768) from the frozen base, in dialogue order
H, meta = encode_dialogues(model, frames, embeddings=E, device="cpu")
```

The package lives at
[`packages/contextual-turn-embeddings`](https://github.com/jumafernandez/doctorado-unsl)
in the project monorepo, together with the partition, the training and evaluation
scripts, the seeds and the commands that regenerate every table.

## Base turn embeddings

The precomputed $e_t$ vectors used as input are published separately as datasets:
`jumafernandez/d2f-turn-embeddings-*`.

## Citation

Fernández, J. M., Errecalde, M., & Burdisso, S. *TRACE: Learning the Update for
Contextual Turn Representations in Task-Oriented Dialogue.* (under review)