data/dataset/README.md

---
license: cc-by-4.0
task_categories:
- translation
- text-generation
language:
- fr
- en
- it
- sk
tags:
- llm
- translation
- back-translation
- semantic-drift
- embeddings
- openrouter
pretty_name: LLM repeated back-translation trajectories
---

# LLM repeated back-translation trajectories

What happens to a text when it is translated back and forth repeatedly by an LLM?

This exploratory dataset starts from short French source texts and sends them through different pivot languages, one translation at a time. Each translation step is a new, stateless API call: the model receives only the fixed translation instruction, the target language, and the previous step's text.

The dataset is designed to explore whether repeated translation produces semantic drift, surface rewriting, compression, convergence, model-specific failures, or pivot-language effects.

## Protocol

For a French source and a pivot language:

```text
FR0 → pivot1 → FR2 → pivot3 → FR4 → ... → FR50
```

- `step = 0`: original French source text.
- Odd steps: translation into the pivot language.
- Even steps: translation back into French.
- Metrics are computed on returned-to-French steps only: `2, 4, ..., 50`, against the original `FR0`.
- Each translation is an independent API call; no conversation history is kept.

## Main experiment dimensions

Sources:

- `source_02`: Alexandre Dumas, *Le Comte de Monte-Cristo* excerpt.
- `source_06`: Marcel Proust, *À l'ombre des jeunes filles en fleurs* excerpt.
- `source_07`: French mathematical/pigeonhole-principle text.

Pivot languages:

- English
- Italian
- Slovak

Translation models:

- `openai/gpt-4o-mini`
- `poolside/laguna-s-2.1`
- `thinkingmachines/inkling-small`

Embedding models used for semantic metrics:

- `openai/text-embedding-3-small`
- `voyageai/voyage-4`

## Files

### `translations.jsonl`

Raw translation trajectories. This is the primary dataset file. It contains every step, including odd pivot-language steps and even returned-to-French steps.

Important fields:

- `experiment_id`
- `timestamp`
- `translation_model`
- `provider`
- `pivot_language`
- `language_pair`
- `source_id`
- `step`
- `language` — requested/expected language for that step
- `text` — raw model output, minimally stripped of transport-level newlines only

### `experiments.jsonl`

Run-level metadata:

- model and provider
- generation parameters
- prompt template
- selected source metadata
- maximum translation steps
- pivot language

Local filesystem/platform details are intentionally excluded from the exported dataset.

### `pivot_to_original_metrics.csv`

Combined metrics against the original French text for both embedding models.

One row is roughly:

```text
source × translation_model × pivot_language × step × embedding_model
```

Key metrics:

- `semantic_similarity_to_original`: cosine similarity between embeddings of `FR0` and `FRn`.
- `word_overlap_to_original`: Jaccard similarity of unique lowercased words.
- `edit_similarity_to_original`: normalized word-level edit similarity.
- `word_count_ratio_to_original`: word count at step `n` divided by word count at step `0`.

### `pivot_pairwise_metrics.csv`

Pairwise comparison between outputs produced through different pivot languages at the same source/model/step.

This is useful for studying whether translation converges to a universal representation or to pivot-conditioned basins.

### Embedding-specific metric files

The combined metric files above are also provided split by embedding model:

- `pivot_to_original_metrics_openai_3_small.csv`
- `pivot_to_original_metrics_voyageai_voyage_4.csv`
- `pivot_pairwise_metrics_openai_3_small.csv`
- `pivot_pairwise_metrics_voyageai_voyage_4.csv`

### `embeddings/`

Raw embedding arrays and row indexes:

```text
embeddings/embeddings_openai_3_small.npy
embeddings/embedding_index_openai_3_small.json
embeddings/embeddings_voyageai_voyage_4.npy
embeddings/embedding_index_voyageai_voyage_4.json
```

Embeddings are derived artifacts and are kept separate from raw translations.

### `sources/` and `sources.jsonl`

Human-readable source Markdown files with front matter, plus a JSONL summary of source metadata.

### `known_failures.csv`

Known obvious output failures detected in the raw trajectories. For example, one model/pivot/source configuration produced premature end markers such as `Fin du texte.` / `Koniec textu.`. These records are preserved in `translations.jsonl` because raw outputs are experimental data, but they are flagged here for analysis.

### `openrouter_models.json`

Cached OpenRouter model metadata snapshot, if available, including context limits, max output tokens, and pricing metadata.

## Translation prompt

The same prompt template was used for all translations:

```text
Translate the following text into {target_language}.

Preserve the meaning, level of certainty, terminology, structure, and style as faithfully as possible.

Return only the translated text. Do not add explanations, comments, notes, quotation marks, or formatting not present in the source.

TEXT:
{text}
```

## Preliminary observations

The dataset supports several exploratory observations:

- Some model/pivot combinations show a sharp initial projection followed by a plateau.
- Literary texts can lose substantial surface/style similarity while preserving broad semantic similarity.
- Pivot language matters: different pivots can produce different stable French paraphrases.
- Model choice matters: some models appear more stable than others under repeated translation.
- Some trajectories can fail instruction-following or collapse into end markers; these are preserved and flagged.

These are exploratory observations, not claims of universal behavior.

## Limitations

- The dataset is small and exploratory.
- `language` is the requested language, not automatically detected language.
- Some model outputs may be malformed or in the wrong language.
- Metrics are approximations: embedding similarity does not capture all stylistic or legal/literary nuances.
- Source licensing/provenance should be reviewed carefully before reuse in downstream publications.

## Suggested use

This dataset is useful for:

- studying repeated LLM translation trajectories;
- comparing semantic vs surface-form preservation;
- exploring model and pivot-language effects;
- building visual tools for inspecting raw translation outputs behind metrics.