docs/00-platform-notes.md

00 — Platform notes (Phase 0)

Observed mechanics of the two execution environments this project spans: this workspace and the Kaggle image. Everything here was measured on 2026-09-19, not recalled. Where a claim contradicts prior documentation (including the Kaggle skill's own notes), the contradiction is called out, because a future session will otherwise trust the older statement.

Raw probe JSON lives in the run logs of the kernels listed in memory/ASSETS.md.

---

1. The two environments are not the same machine

	this workspace	Kaggle CPU session	Kaggle GPU session (2xT4)
OS	Windows 11, git-bash	Linux 6.12.90+, glibc 2.35	Linux 6.12.90+, glibc 2.35
Python	3.11 (and 3.14)	3.12.13	3.12.13
Docker image	n/a	gcr.io/kaggle-images/python@sha256:dafd4ce5…c40b9	gcr.io/kaggle-gpu-images/python@sha256:37c64f7d…7d461
torch	—	2.10.0+cpu	2.10.0+cu128
internet	—	yes (needs enableInternet: true)	yes (needs enableInternet: true)
GPU	none	device_count()==0	2x Tesla T4, cc 7.5, 40 SM, 14.56 GiB usable each

Consequence: CPU and GPU sessions are different images, not one image with a GPU attached. The CPU image's torch is a +cpu build, so a script that imports torch.cuda will work on the GPU shape and silently do nothing useful on the CPU shape. Pin these digests in any job that must be reproducible; a "latest" Kaggle image can move under the run.

huggingface_hub is 1.32.0 locally and reported as huggingface-hub 1.11.0 inside the Kaggle image. Local code and in-job code are therefore on different minor versions — an API that exists locally may not exist in the job (create_repo(tags=…) is one such example; it raised TypeError locally). Check the target environment's version before using a new API.

2. Job mechanics that the main run will depend on

• Submission is one call. save_notebook with kernelExecutionType: "SaveAndRunAll" creates and runs. kernelType: "script" and machineShape must both be present or the call fails with a message-less error. machineShape: "GPU" + enableGpu: true is accepted and yields 2xT4 — no T4-specific shape string is needed, and none should be invented.
• newTitle rewrites the slug. Kept newTitle equal to the slug suffix in every probe so the returned ref matched what was requested. Still: poll the returned slug, never the requested one.
• Latency. A 30 s CPU script went submit → COMPLETE in ~85 s; a ~58 s GPU script in ~145 s. So budget **60–90 s of queue+boot per job**, and don't poll sooner than ~60 s. Both figures are single samples from an uncongested account at ~15:20 UTC.
• Where output lands. list_notebook_session_output returns files[] (signed URLs, expiry unknown — download promptly) and log, a JSON array of {stream_name, time, data} where time is seconds since session start, one array element per line. The log carries the whole stdout, so a probe that prints its result as JSON needs no file download. Only files written under /kaggle/working are returned — the probes deliberately wrote nothing to /tmp, because /tmp looks enormous (§4) and is therefore the easiest place to silently lose an artifact.
• Kernel logs are not secret-safe. Anything printed is retained in the kernel revision. No token may be printed, and by extension no token may be in the kernel source, since source and logs are both Kaggle-held.
• CUDA_VISIBLE_DEVICES is unset in both shapes — it is absent from the environment entirely, not set to the string "None". The Kaggle skill documents CUDA_VISIBLE_DEVICES=None for CPU runs; that is wrong for this image (E-002 in memory/ERRORS.md), and code that branches on == "None" will take the GPU branch on a CPU box. Gate on torch.cuda.is_available() or torch.cuda.device_count() instead.

3. Quota accounting

• Readout is seconds: total_time_allowed: "108000s" = 30 h, quota_refresh_time: 2026-09-26T00:00:00Z. Weekly reset is Saturday 00:00 UTC.
• CPU sessions cost zero GPU quota, confirmed: two CPU probes ran and time_used stayed 0s.
• A 2xT4 session charged 66.411 s for 58.4 s of script time → ~1x wall-clock, not 2x. Full reasoning, the caveat, and the second sample pending in memory/QUOTA.md. This is the single most schedule-relevant number measured in Phase 0: at 1x, 30 h/week is 30 wall-clock hours with both cards, not the 15 that §2's "shared across both cards" phrasing suggests.

4. Disk and memory — the assumption in §2 is only half right

Mount	Total	Free	Notes
/kaggle/working, /kaggle/input	19.5 GB	19.5 GB	the small Kaggle disk; only this is retrieved as output
/dev/loop1 → /kaggle/src	20 GB	20 GB	separate loop device
/, /tmp, /usr, /root/.cache (overlay)	7.9 TB	1.1 TB	shared host filesystem, 88 % used by other tenants
/dev/shm	14 GB	14 GB	useful for DataLoader worker traffic
RAM	32 GB (cgroup memory.max = 30 GiB)	31.2 GB available	SwapTotal: 0
CPU	4 vCPU (cpu.max = 400000 100000), Intel Xeon @ 2.20 GHz

Read this carefully before planning the checkpoint cycle. §2 says "Kaggle disk is small", and for /kaggle/working that is exactly true: 19.5 GB. But the root overlay reports ~1 TB free, and a naive optimisation would write caches or staged shards there. Two reasons not to:

1. The overlay is a shared host volume already at 88 %, so "1 TB free" is other tenants' headroom and can disappear mid-run. §3.13's "never let it fill the disk" must be enforced against a number we do not control.
2. Nothing outside /kaggle/working is retrievable when the session ends, so it is not storage in any sense that matters.

Default HF_HOME/datasets cache resolves under /root/.cache, i.e. the overlay, not the 19.5 GB volume. So a training job's apparent free space can be large while /kaggle/working fills up — the failure mode §3.13 warns about is invisible to a naive disk_usage("/") check. Every job must measure the specific filesystem it writes to, and log free space for that path. On the CPU shape the effective budget is ~19.5 GB; treat that as the design number for shards resident at once.

No swap: a tokenizer build or a large in-memory dedup on a 30 GiB cgroup must be sized, not streamed optimistically.

5. Network behaviour, including one thing that does not resolve

Working, measured from inside a GPU and a CPU session: huggingface.co/api/models → 200 in 149 ms; pypi.org/simple/ → 200 in 73 ms; datasets/…/resolve/main/README.md → 200 via api/resolve-cache/…; datasets.load_dataset("wikimedia/wikipedia", "20231101.en", streaming=True) → first row in 7.4 s, unauthenticated. pip install from PyPI works: xxhash in 5.8 s, import verified.

Does not resolve: cdn-lfs.huggingface.co, cdn-lfs-us-1.huggingface.co, xethub.huggingface.co → gaierror. That looked alarming, so probe D tested it directly by pulling 300 MB from two large public parquet files.

Sustained Hub download throughput from inside a session: 72.8 MB/s (wikimedia/wikipedia, 745 MB file) and 88.9 MB/s (HuggingFaceFW/fineweb-edu, 2334 MB file), with 0.8–0.94 s time-to-first-byte. So the non-resolving LFS hostnames do not block bulk transfer — resolve/ redirects land on hosts that do resolve. Ingest of the mix is not network-bound: at a conservative 70 MB/s, one billion tokens (≈3.6 GB of raw text at ~3.6 chars/token, or ~4–6 GB as parquet) downloads in single-digit minutes. Phase 2 should plan shard staging around CPU and disk limits, not bandwidth.

Also measured in the same probe: datasets streaming at 139.5 rows/s ≈ 3.0 MB of text per second single-process, unauthenticated (83 rows/s / 1.8 MB/s cold, ~3 MB/s warm). Unauthenticated Hub access works but the API warns about rate limits (Please set a HF_TOKEN); at multi-hundred-shard scale that is a plausible 429 source, which is an argument for the token-in-job path (D-002) for the data plane even if the checkpoint plane stays local.

Enumerating repo files without a client library: GET https://huggingface.co/api/datasets/<id>/tree/main?recursive=true returns path/size/type and works with plain urllib. It appeared to cap at 1000 entries (fineweb-edu returned exactly 1000), so assume pagination is required for a large repo.

CPU-side ingest throughput, and what it rules out

Measured on the Kaggle CPU shape (4 vCPU Xeon @ 2.2 GHz) with the gpt2 tokenizer over 6 MB of real Wikipedia English, tokenizers 0.22.2 Tokenizer.encode_batch:

cores	tokens/s	hours to tokenize 1B tokens
1	245,660	1.13
2	492,191	0.56
4	660,943	0.42

chars_per_token 4.53 on this corpus. Scaling is near-linear to 2 cores and then sub-linear (2.7x at 4 cores), which is what a GIL-bound parent feeding rayon workers looks like — so a Phase 2 pipeline that shards by process, not by thread, should recover the rest. datasets streaming ran at 151.6 rows/s ≈ 3.26 MB of text per second single-process, unauthenticated.

The conclusion is a negative result, which is the useful kind: tokenizing 1B tokens costs ~0.4 h of the 4-core budget, and at 35–89 MB/s the raw bytes arrive faster than datasets can stream them. Phase 2 is therefore not CPU-bound and not bandwidth-bound — it is bound by dedup memory and by the 30 GiB cgroup with no swap. Budget the design accordingly: the dedup/audit stage must be written to stream and shard against fixed RAM, not to hold a billion sketches, and the only real wall-clock cost in building the mix is the exact-duplicate/n-gram overlap pass. Note also that download throughput varied by 2.5x between identical runs (36.9 → 72.8 → 63.2 MB/s), so it should be measured inside the actual ingest job rather than treated as a constant.

6. Accelerator capability envelope (Turing, cc 7.5)

Established inside a GPU session, not assumed:

• torch.compile works (warm + 20 steps of an MLP in 6.0 s, incl. first-call compile).
• Flash-attention is not available. SDPBackend.FLASH_ATTENTION raises No available kernel. Aborting execution., preceded by Flash attention only supports gpu architectures in the range [sm80, sm121]. Attempting to run on a sm 7.5 gpu. — so this is architectural, not a missing dependency, and installing flash-attn cannot fix it. §3's "never touch Hopper-only kernels" now has a concrete instance.
• EFFICIENT_ATTENTION and MATH SDPA backends both run. Memory-efficient attention is the viable fast path; note torch warns "Memory efficient attention has been runtime disabled" when flash is force-requested, so backend selection must be explicit.
• torch.cuda.get_arch_list() includes sm_75 (also 70, 80, 86, 90, 100, 120) — this torch build does have kernels for the card.
• CUDA 12.8, cuDNN 9.10.2, NCCL 2.27.5 present; triton 3.6.0 present.
• Absent from the image: flash-attn, xformers, trl, deepspeed, bitsandbytes, liger-kernel, evaluate. Present: transformers 5.0.0, datasets 5.0.0, accelerate 1.13.0, tokenizers 0.22.2, safetensors 0.7.0, numpy 2.0.2, pyarrow 24.0.0, peft 0.19.1.
• bf16 is a hardware question, not a torch one: torch.bfloat16 tensors exist, and bf16 on cc 7.5 is at best emulated. Probe C records whether an autocast-bf16 forward even completes and what it costs, so the Phase 1 precision decision is evidence-backed rather than a recitation of "no bf16".
• nvidia-smi has no multiprocessor_count query field (rc=2); use torch.cuda.… get_device_properties(i).multi_processor_count (40/SM) instead.

7. fp16 training trap found by accident

Probe B ran a small Llama with weights cast to fp16 (model.cuda().to(torch.float16)) under autocast with a plain AdamW and no GradScaler: every loss came back NaN in 65 steps. That is the classic fp16 underflow/overflow signature, and it is exactly the kind of bug that would otherwise be discovered 10 hours into the main run. Probe C re-runs the throughput measurements with the correct recipe — fp32 master weights + torch.autocast(fp16) + torch.amp.GradScaler("cuda") + grad clipping — and asserts finiteness, because on T4 there is no bf16 fallback: fp16 with loss scaling is the only mixed-precision option, so getting it right is not optional.

8. What Phase 0 has not yet proven

Be honest about the boundary of this gate:

• Hub writes from inside a Kaggle job — never attempted. No credential path chosen (D-002 open).
• NCCL across these two T4s — still unproven; probe B's attempt was defeated by its own parser, and one failed all_reduce is not evidence either way. Probe C is fixing that now.
• Checkpoint upload → independent verification → local delete → free-space-confirmed (§3.13) — untouched; that is a Phase 3 gate, and it needs the real 100M-model sizing arithmetic first.
• Cold resume on a fresh instance with an empty disk — Phase 3.
• Real session caps: sessionTimeoutSeconds was accepted and no session ran long enough to hit any limit. The actual wall-clock ceiling, idle-timeout behaviour, and preemption policy are unknown, and the multi-week resume plan in Phase 4 depends on them. Probe E (ounce100m-p0e-session-cap) is a 12.5 h CPU heartbeat running now specifically to find this; if Kaggle kills it, the last heartbeat is the answer. Update while writing: the session was still RUNNING at 108 minutes (submitted 15:38Z, checked 17:26Z), so no CPU cap exists below ~1.8 h. That already rules out the worst case for Phase 4 — a session that dies inside an hour — and the build is sized so one source (5–40 min) is the unit of lost work either way.

9. GPU probe C results — the numbers the schedule is built on

Kernel dodosoomro/ounce100m-p0c-fp16-ddp v2, GPU image gcr.io/kaggle-gpu-images/python@sha256:37c64f7d…7d461, 2x Tesla T4, charged 157.704 s of quota.

Probe B's two void measurements are now replaced. Model shape used — deliberately near the likely main-run shape so the arithmetic is about the real run and not a toy: hidden 768, layers 12, heads 12, GQA kv_heads 3, FFN 2048, vocab 50257, tied embeddings, seq 1024 → 112,934,400 parameters, of which 38,597,376 are the (tied) embedding.

measurement	result
fp32 master weights + autocast(fp16) + GradScaler + clip	loss 9.988, finite — the recipe works; probe B's NaN was the recipe, not the hardware
bf16 autocast on cc 7.5	runs (loss 10.966, first fwd 0.36 s) — i.e. emulated, not tensor-cored
tokens/s, 1 GPU, sdpa, bs4×1024	6,478
tokens/s, 1 GPU, eager, bs4×1024	7,239
tokens/s per rank, DDP 2 ranks, sdpa, bs4×1024	5,531 → 11,062 aggregate
DDP scaling efficiency	11,062 / 6,478 = 1.71x (85 % of ideal)
NCCL across these two T4s	works — rc 0, backend nccl, world 2, both ranks finite, 30 steps
peak CUDA memory, bs4×1024, 1 GPU	11.51 GB (sdpa) / 13.79 GB (eager) of 14.56 GB usable
peak CUDA memory, bs4×1024, DDP	8.68 GB per rank
micro-batch ceiling at this shape	bs 8 OOMs (Tried to allocate 1.54 GiB… 520 MiB free)

What each line forces

• eager beat sdpa here (7,239 vs 6,478 tok/s). At seq_len 1024 the memory-efficient kernel's overhead is not repaid, and flash-attention is unavailable on sm 7.5 anyway. Do not assume "flash is the fast one" — Phase 3 must re-measure at the frozen sequence length, because the crossover is length-dependent.
• Memory, not compute, is the binding constraint on batch size. bs4 fits in 11.5 GB of 14.56 GB; bs8 does not fit at all. So global batch must be built with gradient accumulation, not larger micro-batches — which also means the usual "increase batch until full" advice is unavailable here. The fp32 AdamW state for a 100M model (≈1.2 GB) plus fp32 master weights is a fixed floor that does not shrink with batch size.
• 85 % DDP efficiency is good enough to plan on, and is the number the ETA below uses. It was measured on 30 steps with NCCL over PCIe between two T4s; treat it as optimistic-typical, not a guaranteed constant.
• bf16 running is a trap, not a permission. §2 and §8 both rule it out. On Turing it is emulated, so a config that silently selects bf16 will "work" and be slow. Precision must be asserted to be fp16 in the frozen config, not left to a library default.

First ETA, from measurement rather than hope

1e9 tokens ÷ 11,062 tok/s = 90,400 s ≈ 25.1 wall-clock hours at the measured aggregate rate.

Against a 30 h weekly allowance accruing at 1x, the main run fits inside one week of quota on this shape — but with only ~5 h of headroom, which is not enough to survive both checkpoint upload time and the multi-session restarts that §4/Phase 4 assume. Three honest caveats on that number:

1. It is a 30-step measurement on one shape that is over the parameter budget (112.9 M > 110 M), so the frozen config will differ and throughput with it.
2. It excludes checkpoint write/upload/verify cycles, DataLoader stalls, and any tokenisation-time-vs-disk tradeoff in the input pipeline.
3. It assumes sessions can run long enough to be efficient; the cap is unknown until probe E reports.

Planning stance for docs/01-plan.md: treat ~9–11k tok/s aggregate as the working band, size the token target from the low end (~28–31 h for 1 B tokens → exceeds a single week, so plan for a two-week run), and revisit once Gate 3 measures the frozen configuration. Do not let the 25 h figure become the plan's basis, because it leaves no room for the interruption budget that §3.1 explicitly expects to be spent.

10. Credentials and the Kaggle-side secret store

dodosoomro/ounce100m-p1-credential-path (CPU, read-only) established:

• Internet-enabled Kaggle sessions arrive pre-authenticated against the Kaggle API. The kaggle CLI 2.0.2 is preinstalled, and kaggle config view reports username: dodosoomro, auth_method: ACCESS_TOKEN, config from /root/.config/kaggle — with no kaggle.json written by us. kaggle kernels list --mine returns real rows, so the principal is genuinely usable, not merely present. Injected env vars: KAGGLE_API_V1_TOKEN (32 ch), KAGGLE_DATA_PROXY_TOKEN (473), KAGGLE_USER_SECRETS_TOKEN (245).
• No Hugging Face credential exists in the container: HF_TOKEN and HUGGING_FACE_HUB_TOKEN are both absent, HF_HOME unset, ~/.cache/huggingface does not exist. An anonymous Hub write is correctly rejected — POST /api/models/…/commit/main → 401 Unauthorized — while anonymous reads work. So jobs can pull data freely and cannot push anything until given a token.
• /kaggle/input is empty and /kaggle/input/.secrets/ does not exist, so Kaggle's UI-configured "user input / secrets" facility is unavailable: enabling it is a web-UI action, and there are no locally usable Kaggle credentials to do it with (~/.kaggle absent on this machine; Kaggle auth lives in the MCP server and inside the container). That rules out option 1 of D-002 for a concrete reason.
• Consequence: the in-job authenticated Kaggle API is the usable secret store. A job can create a private Kaggle dataset holding the HF token; later jobs mount it via datasetDataSources and read it from /kaggle/input/…, so the token appears in exactly one private kernel revision instead of in every job's source. §2 explicitly sanctions "a secret store", so this is the intended mechanism.

Ordering used, because the failure mode is unrecoverable. kaggle datasets create derives visibility from a metadata field whose name has moved between CLI versions, and an unrecognised key is silently ignored — which would create a public dataset containing the token. So the bootstrap worker (a) creates the dataset with an inert placeholder only, (b) proves privacy by confirming an anonymous download fails, and (c) uploads the credential as a later version only if (b) passes, then re-proves. If anonymous read ever succeeds, the secret is never written and the job says so loudly.

Residual risk, stated: kernel p1-secret-bootstrap v1 necessarily contains the token inline, and Kaggle retains private kernel revisions — overwrite-after-use does not erase history, and there is no delete-kernel operation in CLI 2.0.2 or in the MCP toolset. This is a strictly smaller exposure than putting the token in every training job and in anything that mirrors kernel source.

11. Pre-existing Kaggle kernels, not ours

kaggle kernels list --mine also returned five kernels predating this project (quick-python-cpu-smoke-test, monte-carlo-cpu-smoke-test, cpu-test, exam-80-monte-carlo, workbuddy-cpu-probe; last run 2026-09-17 → 2026-09-19). They are not touched (§3.10). This also explains E-002: search_notebooks returning {} for this account was the search tool failing, not the account being empty — so an empty search must never be read as "no prior work".

12. Measured notebook session cap: 12.0 h survived, the instance reported 45,000 s

dodosoomro/ounce100m-p0e-session-cap — a CPU kernel that printed one heartbeat every two minutes and nothing else, costing zero GPU quota, left running since 2026-09-19T15:38Z purely as a measurement.

CAP_START {"epoch": 1789832305.41, "host": "096ef365b02e", "pid": 7, "limit": 45000}
HB 1   elapsed=0      free_GB=19.5 mem_avail_MB=31204 load=0.41 net=204
HB 361 elapsed=43204  free_GB=19.5 mem_avail_MB=31288 load=0.00 net=204

361 heartbeats, 43,204 s = 12.0 h of continuous running, then the platform took it (CANCEL_ACKNOWLEDGED, not a crash — the last heartbeat is clean and disk/RAM are unchanged at 19.5 GB free and 31.3 GB available). The instance itself reported limit: 45000 s, i.e. 12.5 h, so the kill arrived at ~96 % of its own stated budget.

What this does and does not license:

• It is a CPU measurement. The GPU accelerator quota is tracked separately, and nothing here proves a GPU session is allowed to run 12.5 h. So Phase 4 plans SESSION_GPU_HOURS=6.9 for session 1 — well inside every plausible cap — and can extend later sessions only on evidence from earlier ones.
• It does settle the question that the plan had been hedging since §2 of 04-run-log.md ("the session cap is unmeasured"): an interruption at 6 h is not the platform's limit, so a long session is not doomed, and a --stop-after-steps schedule sized for 6-7 h is conservative rather than necessary.
• It confirms the working volume is stable for the whole duration — free_GB never moved from 19.5, so an 11 GB drift over a session is not a thing to plan for, and the disk floor in the launcher is about the mix and checkpoints, not about slow leaks.