Assumed infrastructure: what do instruction-tuned LLMs take for granted?

Artifacts of a systematic evaluation of a blind spot in current small open-weight language models: the infrastructure they silently assume when giving practical advice (steady electricity, fast internet, international cards, cloud services, cars and delivery), and whether they truly change their plan when a user states different constraints.

This bucket is a convenience snapshot of everything: the model, the evaluation dataset I built, all model outputs, the audit, the code and a notebook. The GitHub repository and the Hugging Face dataset are the versioned, citable copies; this bucket mirrors release v1.0.0, and MANIFEST.sha256 lists a checksum for every file.

---

Question 1: the blind spot

I use AI models regularly, and I keep noticing that their suggestions do not fit the geographical situation and conditions I am in. When I ask for guidance, the plans I get are often the standard ones of the Western world, and of the United States in particular: use a ride-hailing or delivery service, budget in dollars. Each answer then has to be translated by me into something that is actually feasible in where I am or I provide some precisions to the model about the geographical position and some local condition

This made me suspect a bias: before any country or situation is specified, the model may treat the conditions of the United States as the neutral default. Standard benchmarks rarely capture it. They check whether an answer is correct, fluent or culturally informed, not whether it is feasible for the person asking. An answer can be fluent, polite and well structured and still be unusable, and nothing flags it: there is no factual error, no refusal and no hallucination to detect. The failure is silent, it affects exactly the users who would benefit most from practical guidance, and it is hard to see for those who live under the assumed conditions.

The vague suspicion became three precise questions:

1. Default. With no context, which infrastructure does the model assume?
2. Place. Does naming a place change the plan, and how deeply?
3. Constraints. When the user states the constraints (power cuts, slow and expensive data, no international card, mobile money), does the model replace the steps that became infeasible, or only add alternatives next to them and keep the original steps?

The last question matters most. A response that adds "or use mobile money" but still tells the user to stream video lessons has the look of adaptation without its substance.

---

Question 2: systematic evaluation

Initial hypotheses (written down before any model was run)

The design, hypotheses and analysis plan (docs/PREREGISTRATION.md) were committed in the working repository before any real model output existed. The published repository is a clean snapshot of release v1.0.0, so that ordering can be checked from the document and its history of changes, but not from the published commit dates.

• H1, default world: with no context, responses resemble those given for a high-infrastructure place (Boston, Paris, Seoul).
• H2, shallow adaptation: naming a lower-infrastructure place (Ouagadougou, Lagos, Dhaka, rural Burkina Faso) adds some adaptation but does not remove high-infrastructure reliance.
• H3, inference gap: the model plans well under explicitly stated constraints but does not infer them from the place name alone.

Choice of model

Qwen3-4B-Instruct-2507 is the main model. It has 4.02B parameters, so it sits inside the 0.6-6B range and runs in fp16 (about 8 GB) on a free Colab T4. It is a recent, widely used, instruction-tuned open model from a family that also includes much larger open models, and its licence (Apache-2.0) allows redistribution of weights and outputs. To check that the result is not specific to one model, the same controlled scenarios were run on Phi-4-mini-instruct (3.84B), Llama-3.2-3B-Instruct (3.21B) and Qwen3-1.7B (1.72B). Newer small models exist; extending the replication to them is the natural next step. Runs used two Kaggle T4 GPUs; the Colab notebook runs the same code on one.

What was built

• 120 first-person practical requests: 90 that depend on infrastructure (9 domains: learning, tech, money, business, logistics, health, mobility, shopping, communication) and 30 independent controls. Items 1-3 of each domain are a development split used only to freeze the measurement; items 4-10 form the test split (63 dependent scenarios).
• 10 one-sentence contexts: none; Boston, Paris, Seoul; Ouagadougou, rural Burkina Faso, Lagos, Dhaka; Ouagadougou plus explicit constraints; the same constraints without any place. The constraint profile is an operational persona, not a description of a city or a population.
• 274 real user prompts from OpenAssistant (oasst1), exploratory.
• Three measures: a frozen keyword lexicon (high-infrastructure and adaptation markers), 20 next-token probes with candidate answers fixed in advance, and a blind manual audit with a written rubric (steps that rely on high-infrastructure services without an alternative in the same step; explicit accommodations).
• Analysis: unit = scenario, paired Wilcoxon tests with Holm correction, bootstrap 95% CIs. The lexicon was refined once on the development split, then frozen and tagged in git; the test split was analysed once.

Results

The default world is the United States. When forced to guess the user's country among 8 candidates, Qwen3-4B puts 0.999 of the probability on "the United States" (0.93 with per-token normalisation; the other three models 0.91-0.97). This is a forced choice among 8 candidates, not a calibrated probability over all countries. For practical choices (payment, streaming, cloud, ride-hailing) the probability on high-infrastructure options is 0.72 with no context, 0.53 when the user says they live in Ouagadougou, and 0.36 with explicit constraints.

Explicit constraints change the plan far more than a place name. In the blind audit of Qwen3-4B (24 scenarios), high-infrastructure steps per response fall from 2.46 (no context) to 1.00 (Ouagadougou) and 0.38 (constraints only). Explicit accommodations rise from 0.04 to 0.42 and 4.58.

Most models add; only the largest substitutes.

All four models produce many more accommodations once constraints are stated, so they do read and use them. But for three of the four, the high-infrastructure steps are still there: they add alternatives without removing the steps that the constraints made infeasible (for example, Zoom lessons or an app download advised next to a stated data limit). The confidence intervals of the three smaller models include zero (n = 24), so the claim is the contrast with Qwen3-4B, not a measured increase for any one of them.

Hypotheses. H1 partly supported (the default country and probe results support it; on the lexical measure, high-infrastructure markers for the high places are slightly above no context). H2 mostly not supported as stated: a place name raises adaptation somewhat but it stays small. H3 not supported on the pre-registered lexical measure, but supported by the audit (accommodations 0.42 with a place versus 4.58 with constraints; exploratory). The full verdict table is in docs/RESULTS.md.

Limits, stated plainly

• Four small models, English, one persona per place. Not evidence about frontier models, other languages or real users.
• The audit has a single annotator who is not a resident of the studied places, and blindness is partial (responses naming a place or restating constraints reveal the condition). An independent second-annotation instrument is included but was not completed, so there is no inter-annotator agreement estimate.
• The keyword measure of dependence is weak (Cohen's kappa 0.15 against the manual coding); the audit is the better evidence.
• Many responses are cut at 500 tokens (61% for the main model), so rates describe roughly the first 350 words. Real-prompt results are exploratory.
• A place name is not a measure of a person's access to electricity, internet or payment; the constraint profile is an experimental condition.

---

Question 3: a path forward

Diagnosis. The models can read explicit constraints (accommodations rise sharply in all four), but most do not revise the rest of the plan. My speculation is that training rewards adding helpful alternatives ("or you could...") more than checking whether the whole plan is feasible for the stated user, and that preference data is dominated by users whose infrastructure matches the default.

1. Data: feasibility-conditioned preference pairs.

• Combine scenarios with many infrastructure profiles (power, connectivity, payment, transport, device), written with contributors from different regions rather than one persona.
• For each pair, the chosen response has every step feasible under the profile; the rejected response is the hard negative that this study exposed: it mentions the constraint, adds an alternative, and keeps an infeasible step.
• Include no-context examples where the chosen response states its assumptions or asks one short clarifying question, so that the default is explicit rather than silent.
• Annotate at step level with a rubric like audit/RUBRIC.md, with several annotators and a measured agreement.

2. Training: optimise whole-plan feasibility, not the presence of alternatives.

• Preference optimisation (DPO or similar) on the pairs above, or reinforcement learning with a verifier reward equal to the fraction of steps feasible under the stated constraints, using a judge model that has been calibrated against the human step labels.
• Keep a helpfulness and length term, so that "ask a clarifying question" does not turn into over-asking, and keep infrastructure-independent controls, so that models do not start inventing constraints (the study's spurious-injection check).
• Evaluate on held-out profiles and regions, to test generalisation instead of memorisation of one persona.

3. A cheaper test before any training: plan, check, revise. At inference time, ask the model to list the assumptions of its plan, check each step against the stated constraints, then revise. If this closes most of the gap, the missing ability is checking, not knowledge, which would make the data strategy above the right one; if it does not, the knowledge itself is missing. I would not start with an architectural change.

4. How to know it worked. Re-run this study's audit: the number of high-infrastructure steps should fall when constraints are stated (substitution), without losing helpfulness and without false constraint injection in the controls.

Risk. Inferring a user's constraints from nationality or place would be stereotyping. The target behaviour is to follow stated constraints fully and to make defaults explicit, not to guess who the user is.

---

What is in this bucket

README.md                    this file (answers to questions 1, 2 and 3)
MODEL_SOURCE.md              source, revision and licence of the evaluated model
model/                       a copy of Qwen3-4B-Instruct-2507 at the revision listed in MODEL_SOURCE.md
notebooks/colab_demo.ipynb   runnable demo: load the model, three contexts, probes, recompute headline numbers
data/                        the evaluation dataset: 120 scenarios, 10 contexts, 20 probes, 274 real prompts
results/<model>/             raw generations and probes for the four models (13,096 generations), tables
audit/                       rubric, blind samples, keys, manual codes, second-annotation instrument
figures/                     the two cross-model figures and their data
docs/                        pre-registration, lexicon changelog, full results, data dictionary, licences
src/, scripts/, tests/       code: runner, frozen lexicon, analysis, verification, figures
kaggle/                      builder of the GPU script used for the runs
MANIFEST.sha256              checksum of every file

Reproduce (CPU, a few minutes): pip install -r requirements.txt then python scripts/reproduce.py recomputes every reported number and figure from the outputs and checks them against the shipped tables. Details: docs/REPRODUCE.md.

Licences

Code: MIT. Data, documents and figures created for this study: CC BY 4.0. The real prompts come from OpenAssistant oasst1 (Apache-2.0). The model copy is Qwen3-4B-Instruct-2507 under Apache-2.0. The outputs of Llama-3.2-3B-Instruct are included under the Llama 3.2 Community License (Built with Llama). See docs/THIRD_PARTY_NOTICES.md. Model responses contain errors and hallucinations and must not be used as advice.