Semantic Strands ("Cuerdas"): a pre-registered negative result on learned soft-slot context compression

Technical Report, Revision 5 (16 October 2026; updated 18 October 2026). Not peer reviewed. DOI: 10.5281/zenodo.23448095 Erik Sanchez, independent researcher · ORCID 0009-0001-4908-3084 · sanchezerik836@gmail.com

ES. Les presento los resultados de mi primer reto personal en ciencia computacional. En este punto mi conocimiento todavía es nulo, pero lo compenso con una gran curiosidad. Espero que les sea de utilidad.

EN. These are the results of my first personal challenge in computer science. At this point my knowledge is still close to zero, but I make up for it with a lot of curiosity. I hope you find it useful.

"Cuerdas" (Spanish for "strings") is the internal project name. It has nothing to do with string theory; the English title is Semantic Strands.

Bottom line: this approach did not pay off. A competent compact-text baseline with the same trainable budget beats it. The repository is published so that others can skip this path and reuse the controls that exposed the problem.

• Full report: paper/cuerdas-technical-report.pdf (Markdown source: paper/cuerdas-technical-report.md)
• Citation metadata: CITATION.cff

Note: README files from earlier revisions (1–4) presented the method as a success. That framing is withdrawn. Revision 5 supersedes it.

What was tried

We trained a small encoder ("PackK", K=2, 3,842,952 trainable parameters) to map four-field structured records (action, target, status, window) to 2 learned input vectors ("soft slots") read by a frozen SmolLM2-135M-Instruct (revision 12fd25f77366fa6b3b4b768ec3050bf629380bac; no weights updated, no LoRA). The goal was to replace a 20-token verbal template and so shorten the sequence from 35 to 17 positions per event (counting a shared query prefix and four query tails).

Accuracy is closed-set, first-token, exact match over all four fields. Nothing is generated.

What held

On unseen combinations of known words, the slots match the verbal template on two pre-registered, hash-sealed tests evaluated once (seed means, n=400, 3 seeds):

Sealed test	Verbal template (35 pos.)	K=2 slots (17 pos.)
Iteration 020	99.25%	99.0%
Iteration 021	98.5%	98.7%

What failed

1. Unseen values. On the 021 sealed test (unseen words, names, phrases and multi-token values), accuracy with four unseen values falls to 29%, against 98% for the template. Retraining with a much larger, more varied replacement pool did not help (+2.6 pp, interval includes zero).
2. A fair compact-text baseline wins. With a matched trainable budget (a reparameterized soft prompt of 3,842,368 parameters; same data, steps, loss, teacher and validation-only selection), values separated by commas score 100% on every sealed set, including four unseen values, at 22 positions per event plus 8 shareable prompt positions. Against this baseline the slots save only 2–5 positions per event and lose up to 70.7 pp (four unseen values).
3. The receiver is the bottleneck. Probes trained with replacement words read unseen values from the slots better than the frozen LLM does (for example 97.7% vs 52.6% with two unseen values). The information is in the slots; the receiver does not read it.

Zero-shot compact text fails with this receiver (0–8%), which is why comparing only against zero-shot or verbose text overstates the benefit of soft slots.

Other documented pitfalls: an inflated latency figure from a non-shared prefix (017, corrected in 018), weak probes that made the slots look non-decodable, and a re-saved config.json that silently changed the RoPE base under transformers 4.46.3 (see 019/resultados-019.md and Section 7.6 of the report).

Recommendations for others

Before claiming compression from learned soft slots or soft tokens:

1. Compare against compact text with a matched trainable budget (soft prompt, prefix or LoRA of equal size, same data, steps, loss and selection). Zero-shot or verbose text is not a fair comparison.
2. Count positions honestly, including shared prefixes, query tails and soft-prompt positions.
3. Seal the test: pre-register metrics and decision rules, generate the test with a new seed, hash it and evaluate once.
4. Test unseen values of several kinds (words, names, phrases, multi-token values) and report accuracy by the number of unseen fields.
5. Train probes on the same augmented distribution as the encoder, with shuffled-target controls.
6. Keep a random-weight receiver control, and report per-example predictions with paired bootstrap intervals.
7. Measure latency with a shared KV prefix on every route.
8. Pin model revisions and configuration files.

Scope: one synthetic, closed-vocabulary, four-field task, one 135M receiver, English only. No claim is made beyond it.

Repository contents

Path	What it is
013/	Iteration 013 (CPU, SmolLM2-135M base, smaller universe): turned out to be a lookup table
014-auditoria/	Audit of iteration 014 (run by ChatGPT)
015/ … 018/	Training scripts (run01X.py), exact Kaggle kernels, raw logs (results01X.jsonl) and result notes
019/	Re-evaluation of the 018 checkpoints: per-example predictions, paired bootstrap, random-weight receiver, RoPE-config diagnosis
020/	Pre-registration, sealed test (sealed020.json, SHA-256 4c7737f6…3393), compact-text baselines and stronger probes
021/	Pre-registration, second sealed test (sealed021.json, SHA-256 a86a6a5e…72bc), retraining with a varied pool (c021.py, evalA.py) and matched soft-prompt baselines (evalB.py), logs, predictions and statistics
checkpoints/	binder4_1000_s*.pt (015 teacher), k1_s*.pt (017 K=1), k2_s*.pt (017 K=2, start point for 018 and 021), k2long_s*.pt (018 K=2), k2v_s*.pt (021 K=2 retrained), sp_{values,values_csv,kv}_s*.pt (021 soft-prompt baselines)
make_dataset.py	Regenerates the 014 dataset (standard library only)
paper/	Technical report, revision 5 (PDF, Markdown, Figure 1)

The checkpoints are small encoder state dicts, not standalone language models. They are loaded with the Binder / PackK classes in 018/run018.py (and the soft-prompt class in 021/evalB.py), together with the frozen receiver. Seeds: 17, 29, 43.

How to reproduce

1. Python 3.11+, PyTorch and transformers==4.46.3. The 019 re-evaluation on a Kaggle Tesla T4 recorded PyTorch 2.11.0+cu128, CUDA 12.8, Python 3.13.15, and reproduced every 018 accuracy exactly.
2. Regenerate the dataset: python make_dataset.py --out $SRC/data/expanded.json (verify with --check). The original expanded.json, generated in iteration 014 with ChatGPT, is not redistributed; the script produces the same tuples and split indices.
3. Download the receiver at the pinned revision into $MP. Do not load a copy re-saved by transformers ≥ 5 with transformers 4.46.3: the RoPE base silently falls back to 10000.
4. Training: SRC=... MP=... OD=out python 015/run015.py base binder4:1000, then 016/run016.py, 017/run017.py and 018/run018.py with INP=checkpoints.
5. Evaluations without training: 019/eval019.py and 020/eval020.py (INP must contain the checkpoints, run018.py and the sealed file). Iteration 021: 021/evalA.py (retraining, needs c021.py) and 021/evalB.py (soft-prompt baselines). The kaggle_kernel*.py files show exactly how each was run on a Kaggle T4.

Compute: about 2.6 Kaggle T4 GPU-hours for 015–020 (free tier; Section 4.7 of the report), plus CPU runs for 013–015.

Use of AI tools

AI systems are not authors. Grok Bot (AI assistant) helped design the experiments, wrote and ran most of the code and drafted the report. ChatGPT ran iteration 014, generated its dataset and reviewed earlier revisions. The author set the goals and acceptance criteria, chose what to run next and approved the claims. Details: Section 11 of the report.

License

• Code (scripts, kernels, make_dataset.py) and encoder checkpoints: MIT (see LICENSE).
• Report text (including Figure 1): CC BY 4.0.
• SmolLM2-135M-Instruct is not included and keeps its Apache-2.0 license. The original ChatGPT-generated 014 dataset is not redistributed.

How to cite

Sanchez, E. (2026). Semantic Strands: A Pre-registered Negative Result on Learned Soft-Slot Context Compression — Trained Compact Text Wins. Technical Report, revision 5. Zenodo. https://doi.org/10.5281/zenodo.23448095

DOI: 10.5281/zenodo.23448095

@techreport{sanchez2026cuerdas,
  author      = {Sanchez, Erik},
  title       = {Semantic Strands: A Pre-registered Negative Result on
                 Learned Soft-Slot Context Compression --- Trained
                 Compact Text Wins},
  institution = {Independent researcher},
  type        = {Technical Report},
  number      = {Revision 5},
  year        = {2026},
  month       = oct,
  url         = {https://huggingface.co/Eriksanchezolvera/cuerdas},
  doi         = {10.5281/zenodo.23448095}
}

Contact

Erik Sanchez (independent researcher) · ORCID 0009-0001-4908-3084 · sanchezerik836@gmail.com