| --- |
| license: mit |
| language: |
| - en |
| - es |
| base_model: HuggingFaceTB/SmolLM2-135M-Instruct |
| tags: |
| - soft-prompt |
| - context-compression |
| - prompt-compression |
| - negative-result |
| - pre-registered |
| - research |
| base_model_relation: adapter |
| --- |
| |
| # Semantic Strands ("Cuerdas"): a pre-registered negative result on learned soft-slot context compression |
|
|
| [](https://doi.org/10.5281/zenodo.23448095) |
|
|
| **Technical Report, Revision 5 (16 October 2026; updated 18 October 2026). Not peer reviewed.** DOI: [10.5281/zenodo.23448095](https://doi.org/10.5281/zenodo.23448095) |
| Erik Sanchez, independent researcher · ORCID [0009-0001-4908-3084](https://orcid.org/0009-0001-4908-3084) · sanchezerik836@gmail.com |
|
|
| > **ES.** Les presento los resultados de mi primer reto personal en ciencia computacional. En este punto mi conocimiento todavía es nulo, pero lo compenso con una gran curiosidad. Espero que les sea de utilidad. |
| > |
| > **EN.** These are the results of my first personal challenge in computer science. At this point my knowledge is still close to zero, but I make up for it with a lot of curiosity. I hope you find it useful. |
|
|
| "Cuerdas" (Spanish for "strings") is the internal project name. It has nothing to do with string theory; the English title is *Semantic Strands*. |
|
|
| **Bottom line: this approach did not pay off.** A competent compact-text baseline with the same trainable budget beats it. The repository is published so that others can skip this path and reuse the controls that exposed the problem. |
|
|
| - Full report: [`paper/cuerdas-technical-report.pdf`](paper/cuerdas-technical-report.pdf) (Markdown source: [`paper/cuerdas-technical-report.md`](paper/cuerdas-technical-report.md)) |
| - Citation metadata: [`CITATION.cff`](CITATION.cff) |
|
|
| > Note: README files from earlier revisions (1–4) presented the method as a success. That framing is withdrawn. Revision 5 supersedes it. |
|
|
| ## What was tried |
|
|
| We trained a small encoder ("PackK", K=2, 3,842,952 trainable parameters) to map four-field structured records (action, target, status, window) to **2 learned input vectors ("soft slots")** read by a **frozen** SmolLM2-135M-Instruct (revision `12fd25f77366fa6b3b4b768ec3050bf629380bac`; no weights updated, no LoRA). The goal was to replace a 20-token verbal template and so shorten the sequence from **35 to 17 positions per event** (counting a shared query prefix and four query tails). |
|
|
| Accuracy is closed-set, first-token, exact match over all four fields. Nothing is generated. |
|
|
| ## What held |
|
|
| On unseen combinations of **known** words, the slots match the verbal template on two pre-registered, hash-sealed tests evaluated once (seed means, n=400, 3 seeds): |
|
|
| | Sealed test | Verbal template (35 pos.) | K=2 slots (17 pos.) | |
| |---|---:|---:| |
| | Iteration 020 | 99.25% | 99.0% | |
| | Iteration 021 | 98.5% | 98.7% | |
|
|
| ## What failed |
|
|
| 1. **Unseen values.** On the 021 sealed test (unseen words, names, phrases and multi-token values), accuracy with four unseen values falls to **29%**, against **98%** for the template. Retraining with a much larger, more varied replacement pool did not help (+2.6 pp, interval includes zero). |
| 2. **A fair compact-text baseline wins.** With a matched trainable budget (a reparameterized soft prompt of 3,842,368 parameters; same data, steps, loss, teacher and validation-only selection), **values separated by commas score 100% on every sealed set**, including four unseen values, at 22 positions per event plus 8 shareable prompt positions. Against this baseline the slots save only 2–5 positions per event and lose up to 70.7 pp (four unseen values). |
| 3. **The receiver is the bottleneck.** Probes trained with replacement words read unseen values from the slots better than the frozen LLM does (for example 97.7% vs 52.6% with two unseen values). The information is in the slots; the receiver does not read it. |
|
|
| Zero-shot compact text fails with this receiver (0–8%), which is why comparing only against zero-shot or verbose text overstates the benefit of soft slots. |
|
|
| Other documented pitfalls: an inflated latency figure from a non-shared prefix (017, corrected in 018), weak probes that made the slots look non-decodable, and a re-saved `config.json` that silently changed the RoPE base under transformers 4.46.3 (see `019/resultados-019.md` and Section 7.6 of the report). |
|
|
| ## Recommendations for others |
|
|
| Before claiming compression from learned soft slots or soft tokens: |
|
|
| 1. Compare against **compact text with a matched trainable budget** (soft prompt, prefix or LoRA of equal size, same data, steps, loss and selection). Zero-shot or verbose text is not a fair comparison. |
| 2. Count positions honestly, including shared prefixes, query tails and soft-prompt positions. |
| 3. Seal the test: pre-register metrics and decision rules, generate the test with a new seed, hash it and evaluate once. |
| 4. Test unseen values of several kinds (words, names, phrases, multi-token values) and report accuracy by the number of unseen fields. |
| 5. Train probes on the same augmented distribution as the encoder, with shuffled-target controls. |
| 6. Keep a random-weight receiver control, and report per-example predictions with paired bootstrap intervals. |
| 7. Measure latency with a shared KV prefix on every route. |
| 8. Pin model revisions and configuration files. |
|
|
| Scope: one synthetic, closed-vocabulary, four-field task, one 135M receiver, English only. No claim is made beyond it. |
|
|
| ## Repository contents |
|
|
| | Path | What it is | |
| |---|---| |
| | `013/` | Iteration 013 (CPU, SmolLM2-135M base, smaller universe): turned out to be a lookup table | |
| | `014-auditoria/` | Audit of iteration 014 (run by ChatGPT) | |
| | `015/` … `018/` | Training scripts (`run01X.py`), exact Kaggle kernels, raw logs (`results01X.jsonl`) and result notes | |
| | `019/` | Re-evaluation of the 018 checkpoints: per-example predictions, paired bootstrap, random-weight receiver, RoPE-config diagnosis | |
| | `020/` | Pre-registration, sealed test (`sealed020.json`, SHA-256 `4c7737f6…3393`), compact-text baselines and stronger probes | |
| | `021/` | Pre-registration, second sealed test (`sealed021.json`, SHA-256 `a86a6a5e…72bc`), retraining with a varied pool (`c021.py`, `evalA.py`) and matched soft-prompt baselines (`evalB.py`), logs, predictions and statistics | |
| | `checkpoints/` | `binder4_1000_s*.pt` (015 teacher), `k1_s*.pt` (017 K=1), `k2_s*.pt` (017 K=2, start point for 018 and 021), `k2long_s*.pt` (018 K=2), `k2v_s*.pt` (021 K=2 retrained), `sp_{values,values_csv,kv}_s*.pt` (021 soft-prompt baselines) | |
| | `make_dataset.py` | Regenerates the 014 dataset (standard library only) | |
| | `paper/` | Technical report, revision 5 (PDF, Markdown, Figure 1) | |
|
|
| The checkpoints are small encoder state dicts, **not** standalone language models. They are loaded with the `Binder` / `PackK` classes in `018/run018.py` (and the soft-prompt class in `021/evalB.py`), together with the frozen receiver. Seeds: 17, 29, 43. |
|
|
| ## How to reproduce |
|
|
| 1. Python 3.11+, PyTorch and `transformers==4.46.3`. The 019 re-evaluation on a Kaggle Tesla T4 recorded PyTorch 2.11.0+cu128, CUDA 12.8, Python 3.13.15, and reproduced every 018 accuracy exactly. |
| 2. Regenerate the dataset: `python make_dataset.py --out $SRC/data/expanded.json` (verify with `--check`). The original `expanded.json`, generated in iteration 014 with ChatGPT, is **not redistributed**; the script produces the same tuples and split indices. |
| 3. Download the receiver at the pinned revision into `$MP`. Do **not** load a copy re-saved by transformers ≥ 5 with transformers 4.46.3: the RoPE base silently falls back to 10000. |
| 4. Training: `SRC=... MP=... OD=out python 015/run015.py base binder4:1000`, then `016/run016.py`, `017/run017.py` and `018/run018.py` with `INP=checkpoints`. |
| 5. Evaluations without training: `019/eval019.py` and `020/eval020.py` (`INP` must contain the checkpoints, `run018.py` and the sealed file). Iteration 021: `021/evalA.py` (retraining, needs `c021.py`) and `021/evalB.py` (soft-prompt baselines). The `kaggle_kernel*.py` files show exactly how each was run on a Kaggle T4. |
|
|
| Compute: about 2.6 Kaggle T4 GPU-hours for 015–020 (free tier; Section 4.7 of the report), plus CPU runs for 013–015. |
|
|
| ## Use of AI tools |
|
|
| AI systems are not authors. Grok Bot (AI assistant) helped design the experiments, wrote and ran most of the code and drafted the report. ChatGPT ran iteration 014, generated its dataset and reviewed earlier revisions. The author set the goals and acceptance criteria, chose what to run next and approved the claims. Details: Section 11 of the report. |
|
|
| ## License |
|
|
| - **Code** (scripts, kernels, `make_dataset.py`) and **encoder checkpoints**: MIT (see `LICENSE`). |
| - **Report text** (including Figure 1): CC BY 4.0. |
| - SmolLM2-135M-Instruct is not included and keeps its Apache-2.0 license. The original ChatGPT-generated 014 dataset is not redistributed. |
|
|
| ## How to cite |
|
|
| > Sanchez, E. (2026). *Semantic Strands: A Pre-registered Negative Result on Learned Soft-Slot Context Compression — Trained Compact Text Wins.* Technical Report, revision 5. Zenodo. https://doi.org/10.5281/zenodo.23448095 |
|
|
| DOI: [10.5281/zenodo.23448095](https://doi.org/10.5281/zenodo.23448095) |
|
|
| ```bibtex |
| @techreport{sanchez2026cuerdas, |
| author = {Sanchez, Erik}, |
| title = {Semantic Strands: A Pre-registered Negative Result on |
| Learned Soft-Slot Context Compression --- Trained |
| Compact Text Wins}, |
| institution = {Independent researcher}, |
| type = {Technical Report}, |
| number = {Revision 5}, |
| year = {2026}, |
| month = oct, |
| url = {https://huggingface.co/Eriksanchezolvera/cuerdas}, |
| doi = {10.5281/zenodo.23448095} |
| } |
| ``` |
|
|
| ## Contact |
|
|
| Erik Sanchez (independent researcher) · ORCID [0009-0001-4908-3084](https://orcid.org/0009-0001-4908-3084) · sanchezerik836@gmail.com |
|
|
