| --- |
| license: mit |
| tags: |
| - executorch |
| - xnnpack |
| - pte |
| - on-device |
| - object-detection |
| base_model: |
| - microsoft/table-transformer-structure-recognition |
| --- |
| # Table Transformer — ExecuTorch (find tables, then read their structure) |
|
|
| Two models that pair. **`detection`** finds tables on a page; **`structure`** takes a |
| cropped table and returns its rows, columns, column header and spanning cells. Both are |
| DETR with a ResNet-18 backbone and 125 object queries, 28.8M parameters each. |
|
|
| ``` |
| detect_<H>x<W> pixel_values (1, 3, H, W) fp32 |
| -> logits (1, 125, C+1) fp32, boxes (1, 125, 4) fp32 |
| ``` |
|
|
| - **Files**: `table_transformer_detection_xnnpack_fp32.pte` — **115.8 MB**, |
| `table_transformer_structure_xnnpack_fp32.pte` — **115.9 MB**, three methods each |
| - **Source**: [microsoft/table-transformer-detection](https://huggingface.co/microsoft/table-transformer-detection) and [microsoft/table-transformer-structure-recognition](https://huggingface.co/microsoft/table-transformer-structure-recognition) |
| - **License**: MIT |
| - **Classes**: detection — `table`, `table rotated`. structure — `table`, |
| `table column`, `table row`, `table column header`, `table projected row header`, |
| `table spanning cell` |
|
|
| Boxes come out as DETR always emits them: `(cx, cy, w, h)` **normalised to the input**, |
| so mapping them back to your own page is two lines of arithmetic and does not depend on |
| which rung produced them. |
|
|
| ## The ladder, and why there is one |
|
|
| Each file carries three input sizes: |
|
|
| | rung | for | |
| |---|---| |
| | `detect_667x1000` | landscape — a wide table crop | |
| | `detect_1000x800` | portrait — a page | |
| | `detect_800x800` | square-ish | |
|
|
| **Resize to the rung nearest your aspect ratio.** Do not pad to a square, and do not |
| squash: both were measured against the reference running at its own size, scoring the |
| detections by matched IoU — |
|
|
| | what the caller does | worst matched IoU | |
| |---|---| |
| | resize to the size the processor would have chosen | **1.0000** | |
| | resize to 800x800 (aspect squashed) | 0.8644 | |
| | pad to 1000x1000 with the correct `pixel_mask` | 0.2918 | |
| | pad to 1000x1000 with an all-ones mask | 0.2220, and three detections invented | |
| | resize to 1000x1000 | 0.2095 | |
|
|
| The padded-canvas trick that works for this shelf's audio encoders does not work here, |
| which is why this is a ladder rather than one padded window. Methods in one `.pte` share |
| their constants, so the three rungs cost **0.3 MB** over one: a single method is 115.6 MB |
| and three are 115.9 MB. |
|
|
| ## Running it |
|
|
| **1. Preprocess.** ImageNet mean/std, bilinear resize to the rung, `(1, 3, H, W)`: |
|
|
| ```python |
| mean, std = processor.image_mean, processor.image_std |
| x = (np.asarray(image.resize((W, H))) / 255.0 - mean) / std |
| pixel_values = torch.from_numpy(np.ascontiguousarray(x.transpose(2, 0, 1)))[None] |
| ``` |
|
|
| The copy is deliberate — ExecuTorch reads strides as contiguous whatever the tensor says. |
|
|
| **2. Post-process, outside the graph.** Softmax over the class axis, drop the last |
| column (the "no object" class), keep what clears your threshold: |
|
|
| ```python |
| scores = logits.softmax(-1)[0, :, :-1] |
| best = scores.max(-1) |
| keep = best.values > 0.7 |
| ``` |
|
|
| The threshold is your policy rather than the model's, which is why it is not baked in. |
|
|
| **3. Chain them** for a full page: `detection` to find the table, crop it with a small |
| margin, then `structure` on the crop. |
|
|
| ## Verification |
|
|
| Correlation over the raw output is not the unit this model is used in — 125 queries are |
| mostly the no-object class. The gate is the detections: both arms at the same rung, and |
| for every box the reference found, the best same-label box the build offers. |
|
|
| | model | eager finds (at 0.7) | `.pte` finds | worst matched IoU | |
| |---|---|---|---| |
| | structure | 17 — 9 rows, 4 columns, 1 column header, 2 spanning cells, 1 table | 17 | **1.0000** | |
| | detection | 1 — the table | 1 | **1.0000** | |
|
|
| The gate image is a rendered table, which is what a table in a document is. The |
| reference genuinely reads it: nine rows and four columns of a seven-row, four-column |
| table, plus the shaded header. |
|
|
| ## Speed |
|
|
| Mac arm64, median of 5, at the `667x1000` rung — a reference point for relative cost, |
| not a device number. |
|
|
| | model | `.pte` | torch eager fp32 | |
| |---|---|---| |
| | structure | **44.1 ms** | 66.5 ms | |
| | detection | **37.8 ms** | 63.4 ms | |
|
|
| Faster than eager, which is not the usual result on this shelf and follows from |
| delegation: **90.6%** of the ops run on XNNPACK, in 32 subgraphs. |
|
|
| ## What is not in these files |
|
|
| **fp32 only.** Reduced-precision builds go through this shelf's single-method harness, |
| which measures parity, delegation and timing per file; these are multi-method bundles |
| and would need that path rebuilt for the ladder. At 115 MB for 28.8M parameters there is |
| less to gain here than for the shelf's larger models, and an unmeasured fp16 build is not |
| one this shelf ships. No Core ML build for the same reason. |
|
|
