COOKBOOK.md

# ApplyCRs Cookbook — Conditions for the Application to Work

This document lists the **preconditions** the input documents (CRs, TSs, the Excel
contribution list) and the **environment** must satisfy for the pipeline to apply CRs
correctly. Every rule here comes from a real failure we diagnosed and fixed. When a
CR doesn't apply cleanly, check it against this list *first* — most "bugs" are actually
an input that violates one of these conditions.

The pipeline is: `orchestrate_cr.py` → `fetch_crs.py` (parse Excel, download) →
`cr_parser.py` (read tracked changes) → `ts_applicator.py` (apply them) →
`finalize_ts.py` (title + Change History).

---

## 1. The Excel contribution list

The list is read by `parse_excel` / `parse_excel_all_accepted` in `fetch_crs.py`.

- The sheet must have columns whose headers include **`Status`**, **`Type`**, and a
  submitter column (`SubmittedBy` / similar). Column detection is header-name based.
- Only rows with **`Status == "Accepted"`** and **`Type == CR`** are processed. Anything
  else (Agreed, Revised, Noted, Withdrawn, discussion documents) is silently skipped —
  by design.
- **Submitter filter:**
  - `apply-cr` skill: pass the expert's name → only that person's CRs are processed.
  - `apply-cr-loop` skill: pass an **empty string `""`** → *all* Accepted CRs, any author.
    Never substitute a name here even if the user mentions one.
- Both `.xls` (via `xlrd`) and `.xlsx` are supported.

---

## 2. The CR document — cover page (how the CR identifies its target)

The target TS spec number and version are read from the CR's **cover table** by
`parse_cr_cover` and `extract_cr_metadata`. This is the single most important thing
the CR must get right — if the cover page can't be parsed, the CR is skipped entirely
with `WARNING: could not parse cover page`.

The cover table is located by scanning **all** tables for one containing the literal
text **`CHANGE REQUEST`** (also accepts `CR`, `CHANGE  REQUEST`). Requirements:

- **Spec number** must appear as a cell in the form `NNN NNN` or `NNN NNN-N`
  (e.g. `102 221`, `102 226-2`) — three digits, a single space, three digits. It is
  found either as an exact-match cell, as the cell **immediately after** `CHANGE REQUEST`,
  or as a substring inside a larger cell (fallback). A non-breaking space (`\xa0`) is
  tolerated and normalised.
- **Current version** must appear in the cell **immediately after** a cell containing
  `Current version:`. Accepted formats: `X.Y`, `X.Y.Z`, optionally `V`-prefixed
  (e.g. `18.3.0`, `V18.3.0`). It is normalised to `X.Y.Z`.
- Other metadata read from the cover table for the Change History row: **Meeting #**,
  **CR number**, **Rev**, **Category (Cat)**, and **Title of change**. The title is read
  from the cell after a `Title` / `Title of change` label; the version from `Curr. vers` /
  `Current version`.

**If the cover page uses a non-standard layout** (label and value not adjacent, spec
number split across cells, missing `CHANGE REQUEST` marker), parsing fails. Fix the CR
cover to the standard ETSI CR-form layout rather than patching the parser for one document.

---

## 3. The downloaded TS — title paragraph and version match

`download_ts` fetches the target TS and **verifies it before accepting it**:

- The first paragraph (title) must contain the spec number (space-insensitive).
- The first paragraph must contain **`V{version}`** exactly matching the cover page.
- The file must be a real DOCX (starts with `PK` zip magic).

Consequences you must know:

- **The ETSI portal sometimes returns the wrong version** (e.g. V17.4.0 when V17.3.0 was
  requested). `download_ts` rejects a version mismatch. When the closest available version
  is used deliberately, `update_title_para` falls back to the *actual* version found in the
  title (via `re.search(r'V(\d+\.\d+\.\d+)', title_text)`) and logs a `NOTE`. Body changes
  are still applied regardless of the version mismatch.
- **Some TS versions have no DOCX on the portal at all** (confirmed: TS 102 268 v7.3.0).
  This is a permanent portal limitation — the CR cannot be applied and must be skipped.

### The TS title paragraph
For the title update (`finalize_ts.py`) to work, **`doc.paragraphs[0]`** must:
- Contain the version string as `V{old_version}`.
- Contain a publication date in the form **`(YYYY-MM)`**. If no `(YYYY-MM)` pattern is
  present, the title update is skipped with `[Title] SKIP — no (YYYY-MM) pattern`
  (this is logged, not an error — body changes still apply).

The new version is computed as **X.(Y+1).0** (`derive_new_version`) and the new date via
the 5-day rule (`compute_pub_date`): the next month if today is within 5 days of it,
otherwise the current month.

### The Change History table
`finalize_ts.py` inserts a new row into the **Change History** table, located by finding
a table whose first row contains the text `Change history` (the merged title cell).
- The table's column layout is auto-detected (standard `date|meeting|uid|cr|rev|cat|title|old_v|new_v`,
  plus two known variants without `old_v` or without `date`).
- **Some TSs legitimately have no Change History table** (confirmed: TS 102 267 has only a
  History/Annex table). The `[Change History] NOT PRESENT` log line is **expected and correct**
  for those — not a bug.

---

## 3b. Multiple CRs on the same TS

This is a first-class, supported case — but it has rules.

**Grouping.** CRs are grouped by **`(spec_number, version)`** (orchestrator Step 3). Two
CRs that target the same spec *and* the same current version are put in one group and
applied together to a single TS. The log header shows `-- TS 102 221 v18.3.0 (2 CR(s): X, Y) --`.

**How a group is applied:**
- Each CR in the group is parsed independently, and all their changes are **concatenated
  into one combined manifest** (in the order the UIDs appear in the group).
- The combined manifest is applied to the TS in a **single pass** — the CRs are *not*
  applied to separate copies and merged afterwards.
- The version bump (**X.(Y+1).0**) and the title update happen **once** for the whole
  group, not once per CR.
- **Each** participating CR adds its **own row** to the Change History table (one row per
  CR, all showing the same old→new version).
- If one CR in the group fails to parse or download, the others still apply; the failure is
  recorded per-UID and the group continues.

**⚠️ Order dependency — the main risk.** Because changes are applied sequentially in UID
order, one CR can invalidate another's anchor:
- If CR-A renames text (e.g. `SCP81` → `SCP8X`) and CR-B anchors on the original text, CR-B
  will fail with `anchor not found` once CR-A has been applied — even though CR-B was valid
  against the pristine TS. This is a **known, expected** interaction (see also §6), not a
  pipeline bug. Verify against the *raw* TS XML before treating it as one.
- For the same reason, two CRs must not edit the **exact same run/paragraph** in conflicting
  ways. If they do, the second edit acts on text the first already changed. Non-overlapping
  changes (the normal case — different sections, different tables) combine cleanly.

**Same spec, different versions = separate groups.** If two CRs cite the same spec but
different `Current version` values, they are **different** `(spec, version)` keys → each is
applied to its own separately-downloaded TS, from its own base version. They are never
merged into a single output document. (This usually indicates the contribution list mixes
CRs written against different baselines; each produces its own `ts_..._was_v...docx`.)

---

## 4. How tracked changes must be made in the CR (the core rules)

The parser (`cr_parser.py`) recognises exactly **four** kinds of change. Understanding
the difference between *inline* and *section-level* changes is the single most important
concept — see the ⚠️ box.

### 4a. Inline text change → `text_replace`
The paragraph stays; only words inside it are wrapped in `w:del` / `w:ins`.
Use this for editing text within a sentence, a cell, or a heading.
```xml
<w:p> <w:del><w:delText>old</w:delText></w:del> <w:ins><w:t>new</w:t></w:ins> </w:p>
```
- A **pure deletion** (del with no adjacent ins) is recorded as `old → ""`.
- Whitespace-only deletions with no adjacent insertion are discarded (Word artefacts).
- A `w:ins` that lives inside `w:rPr` is a **formatting change**, not content — ignored.

### 4b. Whole-paragraph / whole-table replacement → `section_replace`
> ⚠️ **The critical distinction.** When the CR author deletes an entire heading + table
> block and types a new one, the change is tracked at the **structural** level: the
> paragraph-mark carries `w:del`/`w:ins` (inside `w:pPr/w:rPr`), and **every row** of a
> table carries `w:del`/`w:ins` in its `w:trPr`. The parser detects the pattern
> *del-block → (optional empty separator) → ins-block* and stores the raw XML of all those
> elements verbatim, so the applicator transplants them into the TS exactly like a Word
> copy-paste (fonts, cell widths, vMerge, styles, authorship all preserved).
>
> **Never** decompose a section replacement into individual `text_replace` / `row_insert`
> operations. Doing so causes: wrong-table matches, lookup poisoning, reversed row order,
> and a visually wrong result. This is enforced as a hard rule.

So, for a CR author: to replace a table, **delete the whole old table and paragraph and
insert the whole new one** (Word does this naturally when you select-and-retype with
track-changes on). Do *not* hand-edit individual cells if the intent is a full rewrite.

### 4c. New table row → `row_insert`
A row whose `w:trPr` contains `w:ins`, inside an otherwise-stable table. The parser records
the cell text/width/style/vMerge and anchors the new row after the nearest preceding
**stable** row (by its column-0 text).

### 4d. Brand-new paragraph → `para_insert`
A wholly-new paragraph (`w:pPr/w:rPr` has `w:ins`) with no corresponding deletion. Anchored
after the previous stable paragraph.

---

## 5. Headings and section numbering in the CR

Anchoring is **section-aware**: every change is stamped with its `section_number`, and the
applicator restricts its search to that section of the TS. For this to work:

- A CR heading must use a real Word **`Heading N` paragraph style** *and* start with a
  dotted number (`8.6`, `11.1.22.3.1`). Both are required:
  - Text starting with a digit but **not** a heading style (e.g. a bit-description line
    `1 = SMS packing…` in style `B30`) is correctly **ignored** — it will not hijack the
    current section.
  - ETSI stores the number and title in separate runs joined by a `<w:tab/>`
    (`<w:t>8.12</w:t><w:tab/><w:t>Result</w:t>`). The parser reconstructs this with the tab,
    so the heading regex matches. Don't collapse the tab into the number in the CR.
- On the TS side, the **Table of Contents** entries have the same text as real headings but
  use `toc N` styles — they are excluded from section detection by the same heading-style
  requirement. Don't worry about the TOC.

---

## 6. How anchors are matched (why "anchor not found" happens)

`ts_applicator.py` finds the target paragraph with **5-tier matching** (highest confidence
first): exact substring → NBSP-normalised → whitespace-normalised → original-text
(reconstruct pre-change text, skipping `w:ins`) → full-text (del + ins concatenated).
Matches below confidence 0.8 log a `WARN`.

For the anchor text in a CR to be found, it must exist **as readable text** in the target
TS section. Failures and their meaning:

- **`WARN section 'X' not found`** → the declared section isn't in this TS; falls back to a
  global search. Usually fine.
- **`ERROR ... anchor declared in section X but found only in section Y`** → the anchor text
  exists but in the wrong place. Often means the CR's section number is wrong, or two CRs in
  the batch renamed the anchor (see below).
- **`ERROR ... anchor not found in TS`** → the text genuinely isn't there. Common real causes:
  - The anchored content is a **figure, image, or screenshot** — there is no text to match.
    These changes **cannot be automated** and are acceptable skips.
  - The content lives in a **zip-embedded archive**, not inline paragraphs (confirmed: the
    Java API in TS 102 267 Annex A). Any CR anchoring on a Java method signature there will
    always fail. Verify with `zipfile` + `re` before spending fix attempts.
  - **Cross-CR anchor renaming:** if CR-A renames `SCP81` → `SCP8X` and CR-B (same batch)
    anchors on the original `SCP81…`, the anchor no longer exists after CR-A applied. Check
    the raw XML before treating it as a bug.

Special case: a very short `old` text (len < 4, e.g. `'1'`) is ambiguous. When a
`para_context` is available it is used to locate the paragraph first, then `old` is verified
inside it — bypassing false matches on titles.

---

## 7. Terminal Profile tables (TS 102 223) — special rule

TS 102 223 has 40+ Terminal Profile byte tables that **all share the same empty header**
(`['', '', '']` with `b8`/`b7`… labels). `_find_table` matches by header, so `'' in anything`
is always true → it returns the **first** table every time, and a tracked deletion of a
`b8` label poisons subsequent lookups.

- **Never** use `_find_table` for these. Always use `_find_table_by_section(doc, "Thirty
  fifth byte:")`, which walks to the heading paragraph and returns the table right after it.
- This is why a full byte-table rewrite must be a **`section_replace`** (§4b), not scattered
  cell edits.

---

## 8. Formatting rules — the CR is the only authority

When inserting new content, take the formatting from the **CR text exactly as written**;
never copy NOTE/bullet/indent conventions from the TS.
- If the CR writes `NOTE: text` (space after colon), use a space — not a tab.
- Preserve leading/trailing spaces and tabs by setting
  `xml:space="preserve"` on the `<w:t>`, or Word silently trims them.
- Use the internal style **`w:val`** (e.g. `B1`, `Heading5`) when building XML, not the
  python-docx style **name** (`B1+`, `Heading 5`) — they differ. Use `get_style_val()`.
- `modify_para_text()` collapses all runs into one and loses per-run formatting — only use
  it on uniformly-formatted paragraphs. The `section_replace` path clones runs verbatim and
  keeps mixed formatting.

---

## 9. Insertion order and index shifting (implementation invariants)

- **Insertion order trap:** repeated `ref.addnext(x)` on the same reference reverses order.
  Advance the pointer after each insert (`ref = new_element`) or use
  `insert_paras_after()` / `insert_rows_after()` from `docx_helpers.py`.
- **Index shifting:** paragraph/row indices change on every insert or delete. Collect all
  needed object references *first*, then mutate; process higher-index sections before lower.

---

## 10. Environment and operational conditions

- **Proxy** must be set before any network call:
  `export http_proxy="http://185.46.212.90:80" https_proxy="http://185.46.212.90:80"`
- **Windows paths** must be converted to WSL form (`C:\...` → `/mnt/c/...`) before passing
  to any script (`wsl_path` / `to_wsl_path`).
- **File locked by Word:** a `PermissionError` on save means the output DOCX is open in Word.
  Save to `<name>_tmp.docx`, ask the user to close Word, then rename.
- **Always verify after save:** re-open the output and confirm section paragraph counts,
  text, order (especially bullets/rows), and that no stray empty paragraphs were introduced.

---

## 11. What is *not* a bug (acceptable skips)

A `SKIP` or `ERROR` line is acceptable **only** when the log shows the change is:
- a **figure, image, or screenshot** (no text to anchor/apply),
- a **formatting-only** change, or
- content in a **portal-unavailable TS version** or a **zip-embedded archive**.

Everything else — a text/table/row change that didn't apply — is a real failure and should
be traced back to one of the conditions above (most often §2 cover page, §4 change type, or
§6 anchor text).