COOKBOOK.md

ApplyCRs Cookbook — Conditions for the Application to Work

This document lists the preconditions the input documents (CRs, TSs, the Excel contribution list) and the environment must satisfy for the pipeline to apply CRs correctly. Every rule here comes from a real failure we diagnosed and fixed. When a CR doesn't apply cleanly, check it against this list first — most "bugs" are actually an input that violates one of these conditions.

The pipeline is: orchestrate_cr.py → fetch_crs.py (parse Excel, download) → cr_parser.py (read tracked changes) → ts_applicator.py (apply them) → finalize_ts.py (title + Change History).

---

1. The Excel contribution list

The list is read by parse_excel / parse_excel_all_accepted in fetch_crs.py.

• The sheet must have columns whose headers include Status, Type, and a submitter column (SubmittedBy / similar). Column detection is header-name based.
• Only rows with Status == "Accepted" and Type == CR are processed. Anything else (Agreed, Revised, Noted, Withdrawn, discussion documents) is silently skipped — by design.
• Submitter filter:
  • apply-cr skill: pass the expert's name → only that person's CRs are processed.
  • apply-cr-loop skill: pass an empty string "" → all Accepted CRs, any author. Never substitute a name here even if the user mentions one.
• Both .xls (via xlrd) and .xlsx are supported.

---

2. The CR document — cover page (how the CR identifies its target)

The target TS spec number and version are read from the CR's cover table by parse_cr_cover and extract_cr_metadata. This is the single most important thing the CR must get right — if the cover page can't be parsed, the CR is skipped entirely with WARNING: could not parse cover page.

The cover table is located by scanning all tables for one containing the literal text CHANGE REQUEST (also accepts CR, CHANGE REQUEST). Requirements:

• Spec number must appear as a cell in the form NNN NNN or NNN NNN-N (e.g. 102 221, 102 226-2) — three digits, a single space, three digits. It is found either as an exact-match cell, as the cell immediately after CHANGE REQUEST, or as a substring inside a larger cell (fallback). A non-breaking space (\xa0) is tolerated and normalised.
• Current version must appear in the cell immediately after a cell containing Current version:. Accepted formats: X.Y, X.Y.Z, optionally V-prefixed (e.g. 18.3.0, V18.3.0). It is normalised to X.Y.Z.
• Other metadata read from the cover table for the Change History row: Meeting #, CR number, Rev, Category (Cat), and Title of change. The title is read from the cell after a Title / Title of change label; the version from Curr. vers / Current version.

If the cover page uses a non-standard layout (label and value not adjacent, spec number split across cells, missing CHANGE REQUEST marker), parsing fails. Fix the CR cover to the standard ETSI CR-form layout rather than patching the parser for one document.

---

3. The downloaded TS — title paragraph and version match

download_ts fetches the target TS and verifies it before accepting it:

• The first paragraph (title) must contain the spec number (space-insensitive).
• The first paragraph must contain V{version} exactly matching the cover page.
• The file must be a real DOCX (starts with PK zip magic).

Consequences you must know:

• The ETSI portal sometimes returns the wrong version (e.g. V17.4.0 when V17.3.0 was requested). download_ts rejects a version mismatch. When the closest available version is used deliberately, update_title_para falls back to the actual version found in the title (via re.search(r'V(\d+\.\d+\.\d+)', title_text)) and logs a NOTE. Body changes are still applied regardless of the version mismatch.
• Some TS versions have no DOCX on the portal at all (confirmed: TS 102 268 v7.3.0). This is a permanent portal limitation — the CR cannot be applied and must be skipped.

The TS title paragraph

For the title update (finalize_ts.py) to work, doc.paragraphs[0] must:

• Contain the version string as V{old_version}.
• Contain a publication date in the form (YYYY-MM). If no (YYYY-MM) pattern is present, the title update is skipped with [Title] SKIP — no (YYYY-MM) pattern (this is logged, not an error — body changes still apply).

The new version is computed as X.(Y+1).0 (derive_new_version) and the new date via the 5-day rule (compute_pub_date): the next month if today is within 5 days of it, otherwise the current month.

The Change History table

finalize_ts.py inserts a new row into the Change History table, located by finding a table whose first row contains the text Change history (the merged title cell).

• The table's column layout is auto-detected (standard date|meeting|uid|cr|rev|cat|title|old_v|new_v, plus two known variants without old_v or without date).
• Some TSs legitimately have no Change History table (confirmed: TS 102 267 has only a History/Annex table). The [Change History] NOT PRESENT log line is expected and correct for those — not a bug.

---

3b. Multiple CRs on the same TS

This is a first-class, supported case — but it has rules.

Grouping. CRs are grouped by (spec_number, version) (orchestrator Step 3). Two CRs that target the same spec and the same current version are put in one group and applied together to a single TS. The log header shows -- TS 102 221 v18.3.0 (2 CR(s): X, Y) --.

How a group is applied:

• Each CR in the group is parsed independently, and all their changes are concatenated into one combined manifest (in the order the UIDs appear in the group).
• The combined manifest is applied to the TS in a single pass — the CRs are not applied to separate copies and merged afterwards.
• The version bump (X.(Y+1).0) and the title update happen once for the whole group, not once per CR.
• Each participating CR adds its own row to the Change History table (one row per CR, all showing the same old→new version).
• If one CR in the group fails to parse or download, the others still apply; the failure is recorded per-UID and the group continues.

⚠️ Order dependency — the main risk. Because changes are applied sequentially in UID order, one CR can invalidate another's anchor:

• If CR-A renames text (e.g. SCP81 → SCP8X) and CR-B anchors on the original text, CR-B will fail with anchor not found once CR-A has been applied — even though CR-B was valid against the pristine TS. This is a known, expected interaction (see also §6), not a pipeline bug. Verify against the raw TS XML before treating it as one.
• For the same reason, two CRs must not edit the exact same run/paragraph in conflicting ways. If they do, the second edit acts on text the first already changed. Non-overlapping changes (the normal case — different sections, different tables) combine cleanly.

Same spec, different versions = separate groups. If two CRs cite the same spec but different Current version values, they are different (spec, version) keys → each is applied to its own separately-downloaded TS, from its own base version. They are never merged into a single output document. (This usually indicates the contribution list mixes CRs written against different baselines; each produces its own ts_..._was_v...docx.)

---

4. How tracked changes must be made in the CR (the core rules)

The parser (cr_parser.py) recognises exactly four kinds of change. Understanding the difference between inline and section-level changes is the single most important concept — see the ⚠️ box.

4a. Inline text change → text_replace

The paragraph stays; only words inside it are wrapped in w:del / w:ins. Use this for editing text within a sentence, a cell, or a heading.

<w:p> <w:del><w:delText>old</w:delText></w:del> <w:ins><w:t>new</w:t></w:ins> </w:p>

• A pure deletion (del with no adjacent ins) is recorded as old → "".
• Whitespace-only deletions with no adjacent insertion are discarded (Word artefacts).
• A w:ins that lives inside w:rPr is a formatting change, not content — ignored.

4b. Whole-paragraph / whole-table replacement → section_replace

⚠️ The critical distinction. When the CR author deletes an entire heading + table block and types a new one, the change is tracked at the structural level: the paragraph-mark carries w:del/w:ins (inside w:pPr/w:rPr), and every row of a table carries w:del/w:ins in its w:trPr. The parser detects the pattern del-block → (optional empty separator) → ins-block and stores the raw XML of all those elements verbatim, so the applicator transplants them into the TS exactly like a Word copy-paste (fonts, cell widths, vMerge, styles, authorship all preserved).

Never decompose a section replacement into individual text_replace / row_insert operations. Doing so causes: wrong-table matches, lookup poisoning, reversed row order, and a visually wrong result. This is enforced as a hard rule.

So, for a CR author: to replace a table, delete the whole old table and paragraph and insert the whole new one (Word does this naturally when you select-and-retype with track-changes on). Do not hand-edit individual cells if the intent is a full rewrite.

4c. New table row → row_insert

A row whose w:trPr contains w:ins, inside an otherwise-stable table. The parser records the cell text/width/style/vMerge and anchors the new row after the nearest preceding stable row (by its column-0 text).

4d. Brand-new paragraph → para_insert

A wholly-new paragraph (w:pPr/w:rPr has w:ins) with no corresponding deletion. Anchored after the previous stable paragraph.

---

5. Headings and section numbering in the CR

Anchoring is section-aware: every change is stamped with its section_number, and the applicator restricts its search to that section of the TS. For this to work:

• A CR heading must use a real Word Heading N paragraph style and start with a dotted number (8.6, 11.1.22.3.1). Both are required:
  • Text starting with a digit but not a heading style (e.g. a bit-description line 1 = SMS packing… in style B30) is correctly ignored — it will not hijack the current section.
  • ETSI stores the number and title in separate runs joined by a <w:tab/> (<w:t>8.12</w:t><w:tab/><w:t>Result</w:t>). The parser reconstructs this with the tab, so the heading regex matches. Don't collapse the tab into the number in the CR.
• On the TS side, the Table of Contents entries have the same text as real headings but use toc N styles — they are excluded from section detection by the same heading-style requirement. Don't worry about the TOC.

---

6. How anchors are matched (why "anchor not found" happens)

ts_applicator.py finds the target paragraph with 5-tier matching (highest confidence first): exact substring → NBSP-normalised → whitespace-normalised → original-text (reconstruct pre-change text, skipping w:ins) → full-text (del + ins concatenated). Matches below confidence 0.8 log a WARN.

For the anchor text in a CR to be found, it must exist as readable text in the target TS section. Failures and their meaning:

• WARN section 'X' not found → the declared section isn't in this TS; falls back to a global search. Usually fine.
• ERROR ... anchor declared in section X but found only in section Y → the anchor text exists but in the wrong place. Often means the CR's section number is wrong, or two CRs in the batch renamed the anchor (see below).
• ERROR ... anchor not found in TS → the text genuinely isn't there. Common real causes:
  • The anchored content is a figure, image, or screenshot — there is no text to match. These changes cannot be automated and are acceptable skips.
  • The content lives in a zip-embedded archive, not inline paragraphs (confirmed: the Java API in TS 102 267 Annex A). Any CR anchoring on a Java method signature there will always fail. Verify with zipfile + re before spending fix attempts.
  • Cross-CR anchor renaming: if CR-A renames SCP81 → SCP8X and CR-B (same batch) anchors on the original SCP81…, the anchor no longer exists after CR-A applied. Check the raw XML before treating it as a bug.

Special case: a very short old text (len < 4, e.g. '1') is ambiguous. When a para_context is available it is used to locate the paragraph first, then old is verified inside it — bypassing false matches on titles.

---

7. Terminal Profile tables (TS 102 223) — special rule

TS 102 223 has 40+ Terminal Profile byte tables that all share the same empty header (['', '', ''] with b8/b7… labels). _find_table matches by header, so '' in anything is always true → it returns the first table every time, and a tracked deletion of a b8 label poisons subsequent lookups.

• Never use _find_table for these. Always use _find_table_by_section(doc, "Thirty fifth byte:"), which walks to the heading paragraph and returns the table right after it.
• This is why a full byte-table rewrite must be a section_replace (§4b), not scattered cell edits.

---

8. Formatting rules — the CR is the only authority

When inserting new content, take the formatting from the CR text exactly as written; never copy NOTE/bullet/indent conventions from the TS.

• If the CR writes NOTE: text (space after colon), use a space — not a tab.
• Preserve leading/trailing spaces and tabs by setting xml:space="preserve" on the <w:t>, or Word silently trims them.
• Use the internal style w:val (e.g. B1, Heading5) when building XML, not the python-docx style name (B1+, Heading 5) — they differ. Use get_style_val().
• modify_para_text() collapses all runs into one and loses per-run formatting — only use it on uniformly-formatted paragraphs. The section_replace path clones runs verbatim and keeps mixed formatting.

---

9. Insertion order and index shifting (implementation invariants)

• Insertion order trap: repeated ref.addnext(x) on the same reference reverses order. Advance the pointer after each insert (ref = new_element) or use insert_paras_after() / insert_rows_after() from docx_helpers.py.
• Index shifting: paragraph/row indices change on every insert or delete. Collect all needed object references first, then mutate; process higher-index sections before lower.

---

10. Environment and operational conditions

• Proxy must be set before any network call: export http_proxy="http://185.46.212.90:80" https_proxy="http://185.46.212.90:80"
• Windows paths must be converted to WSL form (C:\... → /mnt/c/...) before passing to any script (wsl_path / to_wsl_path).
• File locked by Word: a PermissionError on save means the output DOCX is open in Word. Save to <name>_tmp.docx, ask the user to close Word, then rename.
• Always verify after save: re-open the output and confirm section paragraph counts, text, order (especially bullets/rows), and that no stray empty paragraphs were introduced.

---

11. What is not a bug (acceptable skips)

A SKIP or ERROR line is acceptable only when the log shows the change is:

• a figure, image, or screenshot (no text to anchor/apply),
• a formatting-only change, or
• content in a portal-unavailable TS version or a zip-embedded archive.

Everything else — a text/table/row change that didn't apply — is a real failure and should be traced back to one of the conditions above (most often §2 cover page, §4 change type, or §6 anchor text).