Buckets:

SavadogoModeste
/

assumed-infrastructure-eval

Files

xet

SavadogoModeste/assumed-infrastructure-eval / docs /DATA_DICTIONARY.md

SavadogoModeste

19 days ago

preview code

download

raw

6.27 kB

# Data dictionary

Paths are relative to the repository root. Files under `results/` and the large audit files live in the Hugging Face dataset; `data/`, `audit/` and `results_summary/` are also in the GitHub repository.

## Study inputs (`data/`)

### `data/scenarios.csv` (120 rows)
| column | type | meaning |
|---|---|---|
| `id` | string | scenario id: domain letter + number 1-10 (e.g. `L3`, `W7`). Numbers 1-3 form the development split, 4-10 the test split. |
| `group` | string | domain: learning, tech, money, business, logistics, health, mobility, shopping, communication (dependent) or wellbeing, people, home (controls) |
| `dependent` | 0/1 | 1 = outcome plausibly depends on infrastructure; 0 = infrastructure-independent control |
| `text` | string | the first-person request, without any location |

### `data/contexts.json`
Keys `C00_none` ... `C09_constraints_only`: the one-sentence context prepended to each request (empty string for `C00`). Groups used in the analysis: *none* = C00; *high* = C01-C03 (Boston, Paris, Seoul); *low* = C04-C07 (Ouagadougou, rural Burkina Faso, Lagos, Dhaka); *spec_loc* = C08; *spec_only* = C09. `_note` states that C08/C09 are an operational persona.

### `data/probes.json`
`context_probes`: 20 probes, each with `id`, `user` (the question), `prefix` (assistant text already written) and `cands` (list of `[continuation, group]`, group H = high-infrastructure default, L = adapted, N = neutral). `default_world_probe`: the "which country do I live in" probe (`PR0_default_country`), run with no context.

### `data/real_prompts.csv` (274 rows)
| column | meaning |
|---|---|
| `id` | `R001`-`R274` |
| `message_id` | OpenAssistant (oasst1) message id |
| `text` | the user prompt, unchanged |
| `dependent_heuristic` | 1 if a fixed keyword list matches the prompt (prompt only, never model output); used for exploratory splits |

## Model outputs (`results/<model>/`, Hugging Face)

### `generations.jsonl` (one JSON object per line)
| field | meaning |
|---|---|
| `set` | `s1` controlled scenarios, `s2` real prompts (main model only) |
| `scenario_id` | scenario id (`s1`) or `R...` id (`s2`) |
| `group`, `dependent`, `split` | domain, dependence flag, `dev`/`test` (all `s2` rows are `test`) |
| `condition` | `C00_none` ... `C09_constraints_only` |
| `template` | `A` (step-by-step plan, under 300 words), `B` (practical, no length instruction), `R` (real prompt as written) |
| `prompt` | the exact user message (context sentence + request + template suffix) |
| `run` | `greedy` or `sample1` (temperature 0.7, top-p 0.8, top-k 20; seed 1000 + batch offset, so the sample depends on batch size and order; main model only) |
| `response` | the model's text (special tokens stripped) |
| `n_new_tokens` | generated tokens before the stop token or the 500-token cap |
| `truncated` | 1 if the 500-token cap was hit before a stop token |

Counts: Qwen3-4B-Instruct-2507 5,896 (2,400 greedy + 2,400 sampled on `s1`, 1,096 greedy on `s2`); Phi-4-mini-instruct, Llama-3.2-3B-Instruct and Qwen3-1.7B 2,400 each (`s1`, greedy).

### `probes.jsonl` (938 rows per model)
`probe` (probe id), `condition`, `cand` (candidate continuation), `group` (H/L/N), `logp` (sum of token log-probabilities of the candidate given the chat prompt and prefix), `n_tok` (candidate token count), `boundary_shift` (1 if tokenisation of the prefix changed when the candidate was appended; 0 throughout). Within each (`probe`, `condition`) the candidate probabilities are renormalised with a softmax over `logp` (pre-registered); a per-token variant is reported as sensitivity.

### `meta.json`
`model`, `revision` (null when not recorded; see `model_revision_note.txt`), `args` (run arguments), `torch`, `transformers`, `platform`, `cuda`, `n_items`, `data_hash` (first 12 hex characters of SHA-256 of the line-ending-normalised input files), `date`, `seconds`.

## Audit (`audit/`)

The blind audit coded responses for two quantities (rubric in `audit/RUBRIC.md`): `n_high_infra_steps` (steps that rely on high-infrastructure services without an alternative in the same step) and `n_accommodations` (explicit accommodations to low-infrastructure conditions). `generic_disclaimer_only(0/1)` was recorded but is 0 for every response and is not used.

| file | rows | meaning |
|---|---|---|
| `audit_blind.csv` | 120 | main audit: id and response as shown to the annotator (condition hidden); code columns are empty |
| `audit_key.csv` | 120 | `audit_id` -> `scenario_id`, `condition` (kept apart from the coding view) |
| `manual_codes.csv` | 120 | the codes: `audit_id`, `n_high_infra_steps`, `n_accommodations`, `generic_disclaimer_only(0/1)`, `note` |
| `audit_coded.csv` | 120 | blind responses merged with the codes |
| `audit_rep_*.csv` | 96 each | same four files for Phi-4-mini-instruct and Qwen3-1.7B (`audit_rep_key.csv` also has `model`) |
| `audit_llama_*.csv`, `manual_codes_llama.csv` | 48 | same for Llama-3.2-3B-Instruct |
| `second_annotation_sample.csv`, `second_annotation_form.html` | 20 | instrument for an independent second annotation of 20 responses (form generated by `scripts/make_annotation_form.py`); **not completed** |

Main audit: 24 test scenarios x conditions C00, C01, C04, C08, C09 (greedy, template A). Replication audits: the same 24 scenarios x C00 and C09 per model. Annotation: single annotator.

## Derived summary files (`results_summary/` on GitHub; `results/*.md` and per-model tables on Hugging Face)
`tables_test*.md` (all pre-registered tables), `cross_model_test_greedy.md` (primary contrasts per model), `cross_model_test_greedy_excl_G9.md` (sensitivity: same contrasts without scenario G9), `audit_stats.md`, `audit_eval.md`, `replication_audit_stats.md`, per-model figures, and `figures/fig_*_data.csv` (the numbers behind the two cross-model figures).
## Redactions in the released copies
E-mail-like strings inside model responses (hallucinated or placeholder addresses of the form `name@organisation.tld`; 24 occurrences in `generations.jsonl` and the audit response files) are replaced by `[email redacted]` because some could coincide with real contacts. No other text was altered. `scripts/verify_release.py` recomputes every reported number from the redacted files and they match.

Xet Storage Details

Size:: 6.27 kB
Xet hash:: 09b71ebe20d42e6e5c608a7c0023111a0300d769ca8179579f1261214ec4712e

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# Data dictionary

	Paths are relative to the repository root. Files under `results/` and the large audit files live in the Hugging Face dataset; `data/`, `audit/` and `results_summary/` are also in the GitHub repository.

	## Study inputs (`data/`)

	### `data/scenarios.csv` (120 rows)
	\| column \| type \| meaning \|
	\|---\|---\|---\|
	\| `id` \| string \| scenario id: domain letter + number 1-10 (e.g. `L3`, `W7`). Numbers 1-3 form the development split, 4-10 the test split. \|
	\| `group` \| string \| domain: learning, tech, money, business, logistics, health, mobility, shopping, communication (dependent) or wellbeing, people, home (controls) \|
	\| `dependent` \| 0/1 \| 1 = outcome plausibly depends on infrastructure; 0 = infrastructure-independent control \|
	\| `text` \| string \| the first-person request, without any location \|

	### `data/contexts.json`
	Keys `C00_none` ... `C09_constraints_only`: the one-sentence context prepended to each request (empty string for `C00`). Groups used in the analysis: none = C00; high = C01-C03 (Boston, Paris, Seoul); low = C04-C07 (Ouagadougou, rural Burkina Faso, Lagos, Dhaka); spec_loc = C08; spec_only = C09. `_note` states that C08/C09 are an operational persona.

	### `data/probes.json`
	`context_probes`: 20 probes, each with `id`, `user` (the question), `prefix` (assistant text already written) and `cands` (list of `[continuation, group]`, group H = high-infrastructure default, L = adapted, N = neutral). `default_world_probe`: the "which country do I live in" probe (`PR0_default_country`), run with no context.

	### `data/real_prompts.csv` (274 rows)
	\| column \| meaning \|
	\|---\|---\|
	\| `id` \| `R001`-`R274` \|
	\| `message_id` \| OpenAssistant (oasst1) message id \|
	\| `text` \| the user prompt, unchanged \|
	\| `dependent_heuristic` \| 1 if a fixed keyword list matches the prompt (prompt only, never model output); used for exploratory splits \|

	## Model outputs (`results/<model>/`, Hugging Face)

	### `generations.jsonl` (one JSON object per line)
	\| field \| meaning \|
	\|---\|---\|
	\| `set` \| `s1` controlled scenarios, `s2` real prompts (main model only) \|
	\| `scenario_id` \| scenario id (`s1`) or `R...` id (`s2`) \|
	\| `group`, `dependent`, `split` \| domain, dependence flag, `dev`/`test` (all `s2` rows are `test`) \|
	\| `condition` \| `C00_none` ... `C09_constraints_only` \|
	\| `template` \| `A` (step-by-step plan, under 300 words), `B` (practical, no length instruction), `R` (real prompt as written) \|
	\| `prompt` \| the exact user message (context sentence + request + template suffix) \|
	\| `run` \| `greedy` or `sample1` (temperature 0.7, top-p 0.8, top-k 20; seed 1000 + batch offset, so the sample depends on batch size and order; main model only) \|
	\| `response` \| the model's text (special tokens stripped) \|
	\| `n_new_tokens` \| generated tokens before the stop token or the 500-token cap \|
	\| `truncated` \| 1 if the 500-token cap was hit before a stop token \|

	Counts: Qwen3-4B-Instruct-2507 5,896 (2,400 greedy + 2,400 sampled on `s1`, 1,096 greedy on `s2`); Phi-4-mini-instruct, Llama-3.2-3B-Instruct and Qwen3-1.7B 2,400 each (`s1`, greedy).

	### `probes.jsonl` (938 rows per model)
	`probe` (probe id), `condition`, `cand` (candidate continuation), `group` (H/L/N), `logp` (sum of token log-probabilities of the candidate given the chat prompt and prefix), `n_tok` (candidate token count), `boundary_shift` (1 if tokenisation of the prefix changed when the candidate was appended; 0 throughout). Within each (`probe`, `condition`) the candidate probabilities are renormalised with a softmax over `logp` (pre-registered); a per-token variant is reported as sensitivity.

	### `meta.json`
	`model`, `revision` (null when not recorded; see `model_revision_note.txt`), `args` (run arguments), `torch`, `transformers`, `platform`, `cuda`, `n_items`, `data_hash` (first 12 hex characters of SHA-256 of the line-ending-normalised input files), `date`, `seconds`.

	## Audit (`audit/`)

	The blind audit coded responses for two quantities (rubric in `audit/RUBRIC.md`): `n_high_infra_steps` (steps that rely on high-infrastructure services without an alternative in the same step) and `n_accommodations` (explicit accommodations to low-infrastructure conditions). `generic_disclaimer_only(0/1)` was recorded but is 0 for every response and is not used.

	\| file \| rows \| meaning \|
	\|---\|---\|---\|
	\| `audit_blind.csv` \| 120 \| main audit: id and response as shown to the annotator (condition hidden); code columns are empty \|
	\| `audit_key.csv` \| 120 \| `audit_id` -> `scenario_id`, `condition` (kept apart from the coding view) \|
	\| `manual_codes.csv` \| 120 \| the codes: `audit_id`, `n_high_infra_steps`, `n_accommodations`, `generic_disclaimer_only(0/1)`, `note` \|
	\| `audit_coded.csv` \| 120 \| blind responses merged with the codes \|
	\| `audit_rep_*.csv` \| 96 each \| same four files for Phi-4-mini-instruct and Qwen3-1.7B (`audit_rep_key.csv` also has `model`) \|
	\| `audit_llama_*.csv`, `manual_codes_llama.csv` \| 48 \| same for Llama-3.2-3B-Instruct \|
	\| `second_annotation_sample.csv`, `second_annotation_form.html` \| 20 \| instrument for an independent second annotation of 20 responses (form generated by `scripts/make_annotation_form.py`); not completed \|

	Main audit: 24 test scenarios x conditions C00, C01, C04, C08, C09 (greedy, template A). Replication audits: the same 24 scenarios x C00 and C09 per model. Annotation: single annotator.

	## Derived summary files (`results_summary/` on GitHub; `results/*.md` and per-model tables on Hugging Face)
	`tables_test.md` (all pre-registered tables), `cross_model_test_greedy.md` (primary contrasts per model), `cross_model_test_greedy_excl_G9.md` (sensitivity: same contrasts without scenario G9), `audit_stats.md`, `audit_eval.md`, `replication_audit_stats.md`, per-model figures, and `figures/fig__data.csv` (the numbers behind the two cross-model figures).

	## Redactions in the released copies
	E-mail-like strings inside model responses (hallucinated or placeholder addresses of the form `name@organisation.tld`; 24 occurrences in `generations.jsonl` and the audit response files) are replaced by `[email redacted]` because some could coincide with real contacts. No other text was altered. `scripts/verify_release.py` recomputes every reported number from the redacted files and they match.