# ApplyCRs Cookbook — Conditions for the Application to Work This document lists the **preconditions** the input documents (CRs, TSs, the Excel contribution list) and the **environment** must satisfy for the pipeline to apply CRs correctly. Every rule here comes from a real failure we diagnosed and fixed. When a CR doesn't apply cleanly, check it against this list *first* — most "bugs" are actually an input that violates one of these conditions. The pipeline is: `orchestrate_cr.py` → `fetch_crs.py` (parse Excel, download) → `cr_parser.py` (read tracked changes) → `ts_applicator.py` (apply them) → `finalize_ts.py` (title + Change History). --- ## 1. The Excel contribution list The list is read by `parse_excel` / `parse_excel_all_accepted` in `fetch_crs.py`. - The sheet must have columns whose headers include **`Status`**, **`Type`**, and a submitter column (`SubmittedBy` / similar). Column detection is header-name based. - Only rows with **`Status == "Accepted"`** and **`Type == CR`** are processed. Anything else (Agreed, Revised, Noted, Withdrawn, discussion documents) is silently skipped — by design. - **Submitter filter:** - `apply-cr` skill: pass the expert's name → only that person's CRs are processed. - `apply-cr-loop` skill: pass an **empty string `""`** → *all* Accepted CRs, any author. Never substitute a name here even if the user mentions one. - Both `.xls` (via `xlrd`) and `.xlsx` are supported. --- ## 2. The CR document — cover page (how the CR identifies its target) The target TS spec number and version are read from the CR's **cover table** by `parse_cr_cover` and `extract_cr_metadata`. This is the single most important thing the CR must get right — if the cover page can't be parsed, the CR is skipped entirely with `WARNING: could not parse cover page`. The cover table is located by scanning **all** tables for one containing the literal text **`CHANGE REQUEST`** (also accepts `CR`, `CHANGE REQUEST`). Requirements: - **Spec number** must appear as a cell in the form `NNN NNN` or `NNN NNN-N` (e.g. `102 221`, `102 226-2`) — three digits, a single space, three digits. It is found either as an exact-match cell, as the cell **immediately after** `CHANGE REQUEST`, or as a substring inside a larger cell (fallback). A non-breaking space (`\xa0`) is tolerated and normalised. - **Current version** must appear in the cell **immediately after** a cell containing `Current version:`. Accepted formats: `X.Y`, `X.Y.Z`, optionally `V`-prefixed (e.g. `18.3.0`, `V18.3.0`). It is normalised to `X.Y.Z`. - Other metadata read from the cover table for the Change History row: **Meeting #**, **CR number**, **Rev**, **Category (Cat)**, and **Title of change**. The title is read from the cell after a `Title` / `Title of change` label; the version from `Curr. vers` / `Current version`. **If the cover page uses a non-standard layout** (label and value not adjacent, spec number split across cells, missing `CHANGE REQUEST` marker), parsing fails. Fix the CR cover to the standard ETSI CR-form layout rather than patching the parser for one document. --- ## 3. The downloaded TS — title paragraph and version match `download_ts` fetches the target TS and **verifies it before accepting it**: - The first paragraph (title) must contain the spec number (space-insensitive). - The first paragraph must contain **`V{version}`** exactly matching the cover page. - The file must be a real DOCX (starts with `PK` zip magic). Consequences you must know: - **The ETSI portal sometimes returns the wrong version** (e.g. V17.4.0 when V17.3.0 was requested). `download_ts` rejects a version mismatch. When the closest available version is used deliberately, `update_title_para` falls back to the *actual* version found in the title (via `re.search(r'V(\d+\.\d+\.\d+)', title_text)`) and logs a `NOTE`. Body changes are still applied regardless of the version mismatch. - **Some TS versions have no DOCX on the portal at all** (confirmed: TS 102 268 v7.3.0). This is a permanent portal limitation — the CR cannot be applied and must be skipped. ### The TS title paragraph For the title update (`finalize_ts.py`) to work, **`doc.paragraphs[0]`** must: - Contain the version string as `V{old_version}`. - Contain a publication date in the form **`(YYYY-MM)`**. If no `(YYYY-MM)` pattern is present, the title update is skipped with `[Title] SKIP — no (YYYY-MM) pattern` (this is logged, not an error — body changes still apply). The new version is computed as **X.(Y+1).0** (`derive_new_version`) and the new date via the 5-day rule (`compute_pub_date`): the next month if today is within 5 days of it, otherwise the current month. ### The Change History table `finalize_ts.py` inserts a new row into the **Change History** table, located by finding a table whose first row contains the text `Change history` (the merged title cell). - The table's column layout is auto-detected (standard `date|meeting|uid|cr|rev|cat|title|old_v|new_v`, plus two known variants without `old_v` or without `date`). - **Some TSs legitimately have no Change History table** (confirmed: TS 102 267 has only a History/Annex table). The `[Change History] NOT PRESENT` log line is **expected and correct** for those — not a bug. --- ## 3b. Multiple CRs on the same TS This is a first-class, supported case — but it has rules. **Grouping.** CRs are grouped by **`(spec_number, version)`** (orchestrator Step 3). Two CRs that target the same spec *and* the same current version are put in one group and applied together to a single TS. The log header shows `-- TS 102 221 v18.3.0 (2 CR(s): X, Y) --`. **How a group is applied:** - Each CR in the group is parsed independently, and all their changes are **concatenated into one combined manifest** (in the order the UIDs appear in the group). - The combined manifest is applied to the TS in a **single pass** — the CRs are *not* applied to separate copies and merged afterwards. - The version bump (**X.(Y+1).0**) and the title update happen **once** for the whole group, not once per CR. - **Each** participating CR adds its **own row** to the Change History table (one row per CR, all showing the same old→new version). - If one CR in the group fails to parse or download, the others still apply; the failure is recorded per-UID and the group continues. **⚠️ Order dependency — the main risk.** Because changes are applied sequentially in UID order, one CR can invalidate another's anchor: - If CR-A renames text (e.g. `SCP81` → `SCP8X`) and CR-B anchors on the original text, CR-B will fail with `anchor not found` once CR-A has been applied — even though CR-B was valid against the pristine TS. This is a **known, expected** interaction (see also §6), not a pipeline bug. Verify against the *raw* TS XML before treating it as one. - For the same reason, two CRs must not edit the **exact same run/paragraph** in conflicting ways. If they do, the second edit acts on text the first already changed. Non-overlapping changes (the normal case — different sections, different tables) combine cleanly. **Same spec, different versions = separate groups.** If two CRs cite the same spec but different `Current version` values, they are **different** `(spec, version)` keys → each is applied to its own separately-downloaded TS, from its own base version. They are never merged into a single output document. (This usually indicates the contribution list mixes CRs written against different baselines; each produces its own `ts_..._was_v...docx`.) --- ## 4. How tracked changes must be made in the CR (the core rules) The parser (`cr_parser.py`) recognises exactly **four** kinds of change. Understanding the difference between *inline* and *section-level* changes is the single most important concept — see the ⚠️ box. ### 4a. Inline text change → `text_replace` The paragraph stays; only words inside it are wrapped in `w:del` / `w:ins`. Use this for editing text within a sentence, a cell, or a heading. ```xml old new ``` - A **pure deletion** (del with no adjacent ins) is recorded as `old → ""`. - Whitespace-only deletions with no adjacent insertion are discarded (Word artefacts). - A `w:ins` that lives inside `w:rPr` is a **formatting change**, not content — ignored. ### 4b. Whole-paragraph / whole-table replacement → `section_replace` > ⚠️ **The critical distinction.** When the CR author deletes an entire heading + table > block and types a new one, the change is tracked at the **structural** level: the > paragraph-mark carries `w:del`/`w:ins` (inside `w:pPr/w:rPr`), and **every row** of a > table carries `w:del`/`w:ins` in its `w:trPr`. The parser detects the pattern > *del-block → (optional empty separator) → ins-block* and stores the raw XML of all those > elements verbatim, so the applicator transplants them into the TS exactly like a Word > copy-paste (fonts, cell widths, vMerge, styles, authorship all preserved). > > **Never** decompose a section replacement into individual `text_replace` / `row_insert` > operations. Doing so causes: wrong-table matches, lookup poisoning, reversed row order, > and a visually wrong result. This is enforced as a hard rule. So, for a CR author: to replace a table, **delete the whole old table and paragraph and insert the whole new one** (Word does this naturally when you select-and-retype with track-changes on). Do *not* hand-edit individual cells if the intent is a full rewrite. ### 4c. New table row → `row_insert` A row whose `w:trPr` contains `w:ins`, inside an otherwise-stable table. The parser records the cell text/width/style/vMerge and anchors the new row after the nearest preceding **stable** row (by its column-0 text). ### 4d. Brand-new paragraph → `para_insert` A wholly-new paragraph (`w:pPr/w:rPr` has `w:ins`) with no corresponding deletion. Anchored after the previous stable paragraph. --- ## 5. Headings and section numbering in the CR Anchoring is **section-aware**: every change is stamped with its `section_number`, and the applicator restricts its search to that section of the TS. For this to work: - A CR heading must use a real Word **`Heading N` paragraph style** *and* start with a dotted number (`8.6`, `11.1.22.3.1`). Both are required: - Text starting with a digit but **not** a heading style (e.g. a bit-description line `1 = SMS packing…` in style `B30`) is correctly **ignored** — it will not hijack the current section. - ETSI stores the number and title in separate runs joined by a `` (`8.12Result`). The parser reconstructs this with the tab, so the heading regex matches. Don't collapse the tab into the number in the CR. - On the TS side, the **Table of Contents** entries have the same text as real headings but use `toc N` styles — they are excluded from section detection by the same heading-style requirement. Don't worry about the TOC. --- ## 6. How anchors are matched (why "anchor not found" happens) `ts_applicator.py` finds the target paragraph with **5-tier matching** (highest confidence first): exact substring → NBSP-normalised → whitespace-normalised → original-text (reconstruct pre-change text, skipping `w:ins`) → full-text (del + ins concatenated). Matches below confidence 0.8 log a `WARN`. For the anchor text in a CR to be found, it must exist **as readable text** in the target TS section. Failures and their meaning: - **`WARN section 'X' not found`** → the declared section isn't in this TS; falls back to a global search. Usually fine. - **`ERROR ... anchor declared in section X but found only in section Y`** → the anchor text exists but in the wrong place. Often means the CR's section number is wrong, or two CRs in the batch renamed the anchor (see below). - **`ERROR ... anchor not found in TS`** → the text genuinely isn't there. Common real causes: - The anchored content is a **figure, image, or screenshot** — there is no text to match. These changes **cannot be automated** and are acceptable skips. - The content lives in a **zip-embedded archive**, not inline paragraphs (confirmed: the Java API in TS 102 267 Annex A). Any CR anchoring on a Java method signature there will always fail. Verify with `zipfile` + `re` before spending fix attempts. - **Cross-CR anchor renaming:** if CR-A renames `SCP81` → `SCP8X` and CR-B (same batch) anchors on the original `SCP81…`, the anchor no longer exists after CR-A applied. Check the raw XML before treating it as a bug. Special case: a very short `old` text (len < 4, e.g. `'1'`) is ambiguous. When a `para_context` is available it is used to locate the paragraph first, then `old` is verified inside it — bypassing false matches on titles. --- ## 7. Terminal Profile tables (TS 102 223) — special rule TS 102 223 has 40+ Terminal Profile byte tables that **all share the same empty header** (`['', '', '']` with `b8`/`b7`… labels). `_find_table` matches by header, so `'' in anything` is always true → it returns the **first** table every time, and a tracked deletion of a `b8` label poisons subsequent lookups. - **Never** use `_find_table` for these. Always use `_find_table_by_section(doc, "Thirty fifth byte:")`, which walks to the heading paragraph and returns the table right after it. - This is why a full byte-table rewrite must be a **`section_replace`** (§4b), not scattered cell edits. --- ## 8. Formatting rules — the CR is the only authority When inserting new content, take the formatting from the **CR text exactly as written**; never copy NOTE/bullet/indent conventions from the TS. - If the CR writes `NOTE: text` (space after colon), use a space — not a tab. - Preserve leading/trailing spaces and tabs by setting `xml:space="preserve"` on the ``, or Word silently trims them. - Use the internal style **`w:val`** (e.g. `B1`, `Heading5`) when building XML, not the python-docx style **name** (`B1+`, `Heading 5`) — they differ. Use `get_style_val()`. - `modify_para_text()` collapses all runs into one and loses per-run formatting — only use it on uniformly-formatted paragraphs. The `section_replace` path clones runs verbatim and keeps mixed formatting. --- ## 9. Insertion order and index shifting (implementation invariants) - **Insertion order trap:** repeated `ref.addnext(x)` on the same reference reverses order. Advance the pointer after each insert (`ref = new_element`) or use `insert_paras_after()` / `insert_rows_after()` from `docx_helpers.py`. - **Index shifting:** paragraph/row indices change on every insert or delete. Collect all needed object references *first*, then mutate; process higher-index sections before lower. --- ## 10. Environment and operational conditions - **Proxy** must be set before any network call: `export http_proxy="http://185.46.212.90:80" https_proxy="http://185.46.212.90:80"` - **Windows paths** must be converted to WSL form (`C:\...` → `/mnt/c/...`) before passing to any script (`wsl_path` / `to_wsl_path`). - **File locked by Word:** a `PermissionError` on save means the output DOCX is open in Word. Save to `_tmp.docx`, ask the user to close Word, then rename. - **Always verify after save:** re-open the output and confirm section paragraph counts, text, order (especially bullets/rows), and that no stray empty paragraphs were introduced. --- ## 11. What is *not* a bug (acceptable skips) A `SKIP` or `ERROR` line is acceptable **only** when the log shows the change is: - a **figure, image, or screenshot** (no text to anchor/apply), - a **formatting-only** change, or - content in a **portal-unavailable TS version** or a **zip-embedded archive**. Everything else — a text/table/row change that didn't apply — is a real failure and should be traced back to one of the conditions above (most often §2 cover page, §4 change type, or §6 anchor text).