Spaces:
Sleeping
Sleeping
| # ApplyCRs Cookbook — Conditions for the Application to Work | |
| This document lists the **preconditions** the input documents (CRs, TSs, the Excel | |
| contribution list) and the **environment** must satisfy for the pipeline to apply CRs | |
| correctly. Every rule here comes from a real failure we diagnosed and fixed. When a | |
| CR doesn't apply cleanly, check it against this list *first* — most "bugs" are actually | |
| an input that violates one of these conditions. | |
| The pipeline is: `orchestrate_cr.py` → `fetch_crs.py` (parse Excel, download) → | |
| `cr_parser.py` (read tracked changes) → `ts_applicator.py` (apply them) → | |
| `finalize_ts.py` (title + Change History). | |
| --- | |
| ## 1. The Excel contribution list | |
| The list is read by `parse_excel` / `parse_excel_all_accepted` in `fetch_crs.py`. | |
| - The sheet must have columns whose headers include **`Status`**, **`Type`**, and a | |
| submitter column (`SubmittedBy` / similar). Column detection is header-name based. | |
| - Only rows with **`Status == "Accepted"`** and **`Type == CR`** are processed. Anything | |
| else (Agreed, Revised, Noted, Withdrawn, discussion documents) is silently skipped — | |
| by design. | |
| - **Submitter filter:** | |
| - `apply-cr` skill: pass the expert's name → only that person's CRs are processed. | |
| - `apply-cr-loop` skill: pass an **empty string `""`** → *all* Accepted CRs, any author. | |
| Never substitute a name here even if the user mentions one. | |
| - Both `.xls` (via `xlrd`) and `.xlsx` are supported. | |
| --- | |
| ## 2. The CR document — cover page (how the CR identifies its target) | |
| The target TS spec number and version are read from the CR's **cover table** by | |
| `parse_cr_cover` and `extract_cr_metadata`. This is the single most important thing | |
| the CR must get right — if the cover page can't be parsed, the CR is skipped entirely | |
| with `WARNING: could not parse cover page`. | |
| The cover table is located by scanning **all** tables for one containing the literal | |
| text **`CHANGE REQUEST`** (also accepts `CR`, `CHANGE REQUEST`). Requirements: | |
| - **Spec number** must appear as a cell in the form `NNN NNN` or `NNN NNN-N` | |
| (e.g. `102 221`, `102 226-2`) — three digits, a single space, three digits. It is | |
| found either as an exact-match cell, as the cell **immediately after** `CHANGE REQUEST`, | |
| or as a substring inside a larger cell (fallback). A non-breaking space (`\xa0`) is | |
| tolerated and normalised. | |
| - **Current version** must appear in the cell **immediately after** a cell containing | |
| `Current version:`. Accepted formats: `X.Y`, `X.Y.Z`, optionally `V`-prefixed | |
| (e.g. `18.3.0`, `V18.3.0`). It is normalised to `X.Y.Z`. | |
| - Other metadata read from the cover table for the Change History row: **Meeting #**, | |
| **CR number**, **Rev**, **Category (Cat)**, and **Title of change**. The title is read | |
| from the cell after a `Title` / `Title of change` label; the version from `Curr. vers` / | |
| `Current version`. | |
| **If the cover page uses a non-standard layout** (label and value not adjacent, spec | |
| number split across cells, missing `CHANGE REQUEST` marker), parsing fails. Fix the CR | |
| cover to the standard ETSI CR-form layout rather than patching the parser for one document. | |
| --- | |
| ## 3. The downloaded TS — title paragraph and version match | |
| `download_ts` fetches the target TS and **verifies it before accepting it**: | |
| - The first paragraph (title) must contain the spec number (space-insensitive). | |
| - The first paragraph must contain **`V{version}`** exactly matching the cover page. | |
| - The file must be a real DOCX (starts with `PK` zip magic). | |
| Consequences you must know: | |
| - **The ETSI portal sometimes returns the wrong version** (e.g. V17.4.0 when V17.3.0 was | |
| requested). `download_ts` rejects a version mismatch. When the closest available version | |
| is used deliberately, `update_title_para` falls back to the *actual* version found in the | |
| title (via `re.search(r'V(\d+\.\d+\.\d+)', title_text)`) and logs a `NOTE`. Body changes | |
| are still applied regardless of the version mismatch. | |
| - **Some TS versions have no DOCX on the portal at all** (confirmed: TS 102 268 v7.3.0). | |
| This is a permanent portal limitation — the CR cannot be applied and must be skipped. | |
| ### The TS title paragraph | |
| For the title update (`finalize_ts.py`) to work, **`doc.paragraphs[0]`** must: | |
| - Contain the version string as `V{old_version}`. | |
| - Contain a publication date in the form **`(YYYY-MM)`**. If no `(YYYY-MM)` pattern is | |
| present, the title update is skipped with `[Title] SKIP — no (YYYY-MM) pattern` | |
| (this is logged, not an error — body changes still apply). | |
| The new version is computed as **X.(Y+1).0** (`derive_new_version`) and the new date via | |
| the 5-day rule (`compute_pub_date`): the next month if today is within 5 days of it, | |
| otherwise the current month. | |
| ### The Change History table | |
| `finalize_ts.py` inserts a new row into the **Change History** table, located by finding | |
| a table whose first row contains the text `Change history` (the merged title cell). | |
| - The table's column layout is auto-detected (standard `date|meeting|uid|cr|rev|cat|title|old_v|new_v`, | |
| plus two known variants without `old_v` or without `date`). | |
| - **Some TSs legitimately have no Change History table** (confirmed: TS 102 267 has only a | |
| History/Annex table). The `[Change History] NOT PRESENT` log line is **expected and correct** | |
| for those — not a bug. | |
| --- | |
| ## 3b. Multiple CRs on the same TS | |
| This is a first-class, supported case — but it has rules. | |
| **Grouping.** CRs are grouped by **`(spec_number, version)`** (orchestrator Step 3). Two | |
| CRs that target the same spec *and* the same current version are put in one group and | |
| applied together to a single TS. The log header shows `-- TS 102 221 v18.3.0 (2 CR(s): X, Y) --`. | |
| **How a group is applied:** | |
| - Each CR in the group is parsed independently, and all their changes are **concatenated | |
| into one combined manifest** (in the order the UIDs appear in the group). | |
| - The combined manifest is applied to the TS in a **single pass** — the CRs are *not* | |
| applied to separate copies and merged afterwards. | |
| - The version bump (**X.(Y+1).0**) and the title update happen **once** for the whole | |
| group, not once per CR. | |
| - **Each** participating CR adds its **own row** to the Change History table (one row per | |
| CR, all showing the same old→new version). | |
| - If one CR in the group fails to parse or download, the others still apply; the failure is | |
| recorded per-UID and the group continues. | |
| **⚠️ Order dependency — the main risk.** Because changes are applied sequentially in UID | |
| order, one CR can invalidate another's anchor: | |
| - If CR-A renames text (e.g. `SCP81` → `SCP8X`) and CR-B anchors on the original text, CR-B | |
| will fail with `anchor not found` once CR-A has been applied — even though CR-B was valid | |
| against the pristine TS. This is a **known, expected** interaction (see also §6), not a | |
| pipeline bug. Verify against the *raw* TS XML before treating it as one. | |
| - For the same reason, two CRs must not edit the **exact same run/paragraph** in conflicting | |
| ways. If they do, the second edit acts on text the first already changed. Non-overlapping | |
| changes (the normal case — different sections, different tables) combine cleanly. | |
| **Same spec, different versions = separate groups.** If two CRs cite the same spec but | |
| different `Current version` values, they are **different** `(spec, version)` keys → each is | |
| applied to its own separately-downloaded TS, from its own base version. They are never | |
| merged into a single output document. (This usually indicates the contribution list mixes | |
| CRs written against different baselines; each produces its own `ts_..._was_v...docx`.) | |
| --- | |
| ## 4. How tracked changes must be made in the CR (the core rules) | |
| The parser (`cr_parser.py`) recognises exactly **four** kinds of change. Understanding | |
| the difference between *inline* and *section-level* changes is the single most important | |
| concept — see the ⚠️ box. | |
| ### 4a. Inline text change → `text_replace` | |
| The paragraph stays; only words inside it are wrapped in `w:del` / `w:ins`. | |
| Use this for editing text within a sentence, a cell, or a heading. | |
| ```xml | |
| <w:p> <w:del><w:delText>old</w:delText></w:del> <w:ins><w:t>new</w:t></w:ins> </w:p> | |
| ``` | |
| - A **pure deletion** (del with no adjacent ins) is recorded as `old → ""`. | |
| - Whitespace-only deletions with no adjacent insertion are discarded (Word artefacts). | |
| - A `w:ins` that lives inside `w:rPr` is a **formatting change**, not content — ignored. | |
| ### 4b. Whole-paragraph / whole-table replacement → `section_replace` | |
| > ⚠️ **The critical distinction.** When the CR author deletes an entire heading + table | |
| > block and types a new one, the change is tracked at the **structural** level: the | |
| > paragraph-mark carries `w:del`/`w:ins` (inside `w:pPr/w:rPr`), and **every row** of a | |
| > table carries `w:del`/`w:ins` in its `w:trPr`. The parser detects the pattern | |
| > *del-block → (optional empty separator) → ins-block* and stores the raw XML of all those | |
| > elements verbatim, so the applicator transplants them into the TS exactly like a Word | |
| > copy-paste (fonts, cell widths, vMerge, styles, authorship all preserved). | |
| > | |
| > **Never** decompose a section replacement into individual `text_replace` / `row_insert` | |
| > operations. Doing so causes: wrong-table matches, lookup poisoning, reversed row order, | |
| > and a visually wrong result. This is enforced as a hard rule. | |
| So, for a CR author: to replace a table, **delete the whole old table and paragraph and | |
| insert the whole new one** (Word does this naturally when you select-and-retype with | |
| track-changes on). Do *not* hand-edit individual cells if the intent is a full rewrite. | |
| ### 4c. New table row → `row_insert` | |
| A row whose `w:trPr` contains `w:ins`, inside an otherwise-stable table. The parser records | |
| the cell text/width/style/vMerge and anchors the new row after the nearest preceding | |
| **stable** row (by its column-0 text). | |
| ### 4d. Brand-new paragraph → `para_insert` | |
| A wholly-new paragraph (`w:pPr/w:rPr` has `w:ins`) with no corresponding deletion. Anchored | |
| after the previous stable paragraph. | |
| --- | |
| ## 5. Headings and section numbering in the CR | |
| Anchoring is **section-aware**: every change is stamped with its `section_number`, and the | |
| applicator restricts its search to that section of the TS. For this to work: | |
| - A CR heading must use a real Word **`Heading N` paragraph style** *and* start with a | |
| dotted number (`8.6`, `11.1.22.3.1`). Both are required: | |
| - Text starting with a digit but **not** a heading style (e.g. a bit-description line | |
| `1 = SMS packing…` in style `B30`) is correctly **ignored** — it will not hijack the | |
| current section. | |
| - ETSI stores the number and title in separate runs joined by a `<w:tab/>` | |
| (`<w:t>8.12</w:t><w:tab/><w:t>Result</w:t>`). The parser reconstructs this with the tab, | |
| so the heading regex matches. Don't collapse the tab into the number in the CR. | |
| - On the TS side, the **Table of Contents** entries have the same text as real headings but | |
| use `toc N` styles — they are excluded from section detection by the same heading-style | |
| requirement. Don't worry about the TOC. | |
| --- | |
| ## 6. How anchors are matched (why "anchor not found" happens) | |
| `ts_applicator.py` finds the target paragraph with **5-tier matching** (highest confidence | |
| first): exact substring → NBSP-normalised → whitespace-normalised → original-text | |
| (reconstruct pre-change text, skipping `w:ins`) → full-text (del + ins concatenated). | |
| Matches below confidence 0.8 log a `WARN`. | |
| For the anchor text in a CR to be found, it must exist **as readable text** in the target | |
| TS section. Failures and their meaning: | |
| - **`WARN section 'X' not found`** → the declared section isn't in this TS; falls back to a | |
| global search. Usually fine. | |
| - **`ERROR ... anchor declared in section X but found only in section Y`** → the anchor text | |
| exists but in the wrong place. Often means the CR's section number is wrong, or two CRs in | |
| the batch renamed the anchor (see below). | |
| - **`ERROR ... anchor not found in TS`** → the text genuinely isn't there. Common real causes: | |
| - The anchored content is a **figure, image, or screenshot** — there is no text to match. | |
| These changes **cannot be automated** and are acceptable skips. | |
| - The content lives in a **zip-embedded archive**, not inline paragraphs (confirmed: the | |
| Java API in TS 102 267 Annex A). Any CR anchoring on a Java method signature there will | |
| always fail. Verify with `zipfile` + `re` before spending fix attempts. | |
| - **Cross-CR anchor renaming:** if CR-A renames `SCP81` → `SCP8X` and CR-B (same batch) | |
| anchors on the original `SCP81…`, the anchor no longer exists after CR-A applied. Check | |
| the raw XML before treating it as a bug. | |
| Special case: a very short `old` text (len < 4, e.g. `'1'`) is ambiguous. When a | |
| `para_context` is available it is used to locate the paragraph first, then `old` is verified | |
| inside it — bypassing false matches on titles. | |
| --- | |
| ## 7. Terminal Profile tables (TS 102 223) — special rule | |
| TS 102 223 has 40+ Terminal Profile byte tables that **all share the same empty header** | |
| (`['', '', '']` with `b8`/`b7`… labels). `_find_table` matches by header, so `'' in anything` | |
| is always true → it returns the **first** table every time, and a tracked deletion of a | |
| `b8` label poisons subsequent lookups. | |
| - **Never** use `_find_table` for these. Always use `_find_table_by_section(doc, "Thirty | |
| fifth byte:")`, which walks to the heading paragraph and returns the table right after it. | |
| - This is why a full byte-table rewrite must be a **`section_replace`** (§4b), not scattered | |
| cell edits. | |
| --- | |
| ## 8. Formatting rules — the CR is the only authority | |
| When inserting new content, take the formatting from the **CR text exactly as written**; | |
| never copy NOTE/bullet/indent conventions from the TS. | |
| - If the CR writes `NOTE: text` (space after colon), use a space — not a tab. | |
| - Preserve leading/trailing spaces and tabs by setting | |
| `xml:space="preserve"` on the `<w:t>`, or Word silently trims them. | |
| - Use the internal style **`w:val`** (e.g. `B1`, `Heading5`) when building XML, not the | |
| python-docx style **name** (`B1+`, `Heading 5`) — they differ. Use `get_style_val()`. | |
| - `modify_para_text()` collapses all runs into one and loses per-run formatting — only use | |
| it on uniformly-formatted paragraphs. The `section_replace` path clones runs verbatim and | |
| keeps mixed formatting. | |
| --- | |
| ## 9. Insertion order and index shifting (implementation invariants) | |
| - **Insertion order trap:** repeated `ref.addnext(x)` on the same reference reverses order. | |
| Advance the pointer after each insert (`ref = new_element`) or use | |
| `insert_paras_after()` / `insert_rows_after()` from `docx_helpers.py`. | |
| - **Index shifting:** paragraph/row indices change on every insert or delete. Collect all | |
| needed object references *first*, then mutate; process higher-index sections before lower. | |
| --- | |
| ## 10. Environment and operational conditions | |
| - **Proxy** must be set before any network call: | |
| `export http_proxy="http://185.46.212.90:80" https_proxy="http://185.46.212.90:80"` | |
| - **Windows paths** must be converted to WSL form (`C:\...` → `/mnt/c/...`) before passing | |
| to any script (`wsl_path` / `to_wsl_path`). | |
| - **File locked by Word:** a `PermissionError` on save means the output DOCX is open in Word. | |
| Save to `<name>_tmp.docx`, ask the user to close Word, then rename. | |
| - **Always verify after save:** re-open the output and confirm section paragraph counts, | |
| text, order (especially bullets/rows), and that no stray empty paragraphs were introduced. | |
| --- | |
| ## 11. What is *not* a bug (acceptable skips) | |
| A `SKIP` or `ERROR` line is acceptable **only** when the log shows the change is: | |
| - a **figure, image, or screenshot** (no text to anchor/apply), | |
| - a **formatting-only** change, or | |
| - content in a **portal-unavailable TS version** or a **zip-embedded archive**. | |
| Everything else — a text/table/row change that didn't apply — is a real failure and should | |
| be traced back to one of the conditions above (most often §2 cover page, §4 change type, or | |
| §6 anchor text). | |