HTS classifier (image and/or text β US HTS code + customs description)
A CPU serving bundle for a demo. It maps a product photo and/or a product title to a 10-digit
US Harmonized Tariff Schedule (HTS) code and returns a β€30-character customs description, alternates,
a confidence score, an abstain flag, and the nearest past customs rulings as justification.
It runs on onnxruntime + tokenizers + numpy (no torch; the reranker's trees are evaluated in numpy).
On 4 CPU threads: under 1 GB RSS, about 30β45 ms per text query including reranking, and about 250 ms with a photo.
Bundle version: r4 + reranker.
Not legal advice. Classification depends on facts a photo or title may not show (material, function, composition). Use the output as a starting point and confirm it against the HTS and with a licensed broker.
How it works
It retrieves and reranks rather than using a softmax classifier:
- Text tower: the jina-clip-v1 text encoder, fully fine-tuned (768-d, mean pooling,
queries up to 256 tokens). The loss is InfoNCE with a learned temperature, pulling queries
toward their HTS leaf description and toward past rulings with the same code.
Related codes (shared prefix) are masked out as false negatives. A small heading-level
term and an anchor to the original model are added. Training ran in two phases on 566,982
query β code pairs, batch 256: 1,500 steps with tariff-tree sibling codes as hard negatives,
then 2,500 steps with negatives mined by the phase-1 model.
It is exported to ONNX with weight-only 8-bit
MatMulNBits(block 128). - Image tower: the frozen jina-clip-v1 vision encoder (weight-only 8-bit ONNX) plus a
768Γ768 linear adapter with a fixed modality-gap shift. The adapter is trained to map photos
into the fine-tuned text space. Photo + title queries use
norm(0.2Β·image + 0.8Β·text). - Index: 20,418 HTS leaf descriptions + 203,714 past ruling snippets, PCA 768β256, per-dimension int8, exact (brute-force) scoring with an int8 GEMM inside onnxruntime.
- Fusion: leaf similarity and votes from the 100 nearest rulings (summed over 4/6/8/10-digit prefixes) are mixed 50/50 into a candidate shortlist.
- Reranker: LightGBM (
lambdarank, 250 trees Γ 7 leaves, monotone constraints) rescores the top 30 candidates with 24 features: fused rank and score, leaf gap, per-level ruling votes, keyword (BM25) match against the leaf text and plain-language aliases, "Other"-bucket and sibling counts, and stated-fact checks (gender, fiber, knit/woven, quantities). It was trained only on 2023 validation rulings plus half of an extra CBP set, using 5-fold out-of-fold training grouped by gold code. Benchmarks and 2024+ test rulings were never in its training data. - Confidence: the reranker's softmax probability mass on the top 6-digit code. The demo reports the digits it is confident in by walking down the tariff tree while one branch holds β₯ 0.6 of the mass. Below that depth it asks a clarifying question.
Files
| file | what |
|---|---|
text.onnx |
fine-tuned text tower, weight-only 8-bit (input input_ids, batch 1, β€256 tokens) |
vision.onnx |
jina-clip-v1 vision tower, weight-only 8-bit (input pixel_values [1,3,224,224]) |
tokenizer.json |
tokenizer for text.onnx |
image_adapter.npz |
W, b, gap of the image adapter |
index.npz |
PCA matrix (768β256) and per-dim int8 scales |
score_leaf.onnx, score_pool.onnx |
int8 GEMM scorers holding the leaf / ruling embeddings |
leaves.parquet |
HTS leaf codes, descriptions, paths (HTS revision downloaded 2026-11-12) |
pool.parquet |
ruling snippets: label (code), source, text |
rerank.json |
reranker trees, feature list, softmax temperature (T = 2), abstain threshold |
rr_aliases.json |
plain-language aliases per leaf, used by the reranker's keyword features |
meta.json |
serving settings (max_tokens) |
Serving code: hts/serve.py and hts/engine.py in the project repository.
Evaluation (shipped path: fused shortlist β reranker, strict hit rule)
A prediction counts as correct only if it matches the gold code to the stated depth. Every query is in the denominator, so a ruling whose gold code is shorter than 10 digits counts as a 10-digit miss. Training uses US rulings dated 2022 or earlier; validation uses 2023; the test sets use 2024 or later. Near-duplicate rulings are merged before scoring. Intervals are 95% Wilson intervals.
| set | n | R@10 | top-1 HS6 | top-1 10-digit |
|---|---|---|---|---|
| US CBP rulings 2024+, full description | 2,855 | 0.890 | 0.658 | 0.415 |
| US CBP rulings 2024+, short description | 2,480 | 0.787 | 0.533 | 0.304 |
| HSCodeComp (product titles, text only) | 632 | 0.752 | 0.503 [0.464, 0.542] | 0.342 [0.306, 0.380] |
| ATLAS test (product titles) | 200 | 0.710 | 0.540 [0.471, 0.608] | 0.335 [0.273, 0.403] |
What the reranker adds over the fused shortlist on the same bundle: +0.6 to +2.5 points top-1 10-digit on the test sets (β0.3 on out-of-fold validation), +2 to +5 points R@10, and fewer clarifying questions. On 500 real listings, questions per answer dropped from 1.04 to 0.73.
Real marketplace listings (500 product pages; gold = the depth at which two independent LLM labelers agreed, usually 4β6 digits; this set was also used for model selection): 0.784 top-1.
Photos:
- ABO product photos (725 items, 29 product types, heading level): image only 0.67 top-1 heading, photo + title 0.89 through the reranker. ABO was also used to tune photo weights.
- 424 real listings with photos: photo + title 0.792 vs title only 0.767 (+13/β2 items, sign test p = .007).
Abstain threshold: abstain when the top HS6 mass is below 0.375. On out-of-fold validation the model answers 56% of queries, with 74% HS6 precision on those answers. This is a probability mass, not a calibrated probability of being correct. Calibration has not been checked on the test sets or on photo inputs.
All results are from a single training seed. Quantization was validated end to end on an earlier model (within about 1 point; top-1 agreement with fp32 0.99). The numbers above are measured through the shipped 8-bit stack.
Data and attribution
Ruling snippets in pool.parquet and the training data come from:
- US CBP CROSS rulings (US government work), via the
orlandowhite/hts-cross-rulingsandfklc/cbp-rulings-past-2012(MIT) datasets. - EU Binding Tariff Information (EBTI) and EU classification regulations, Β© European Union, reused under Commission Decision 2011/833/EU / CC BY 4.0.
- UK Advance Tariff Rulings, Β© Crown copyright, Open Government Licence v3.0.
- Australian Tariff Concession Orders, Β© Commonwealth of Australia (Department of Home Affairs), CC BY 3.0 AU.
- Canada Border Services Agency (CBSA) advance rulings: 1,201 snippets (0.6% of the pool).
- US HTS from the US International Trade Commission.
- About half of the training rows are real ruling text. The rest are LLM-generated or
LLM-labeled: synthetic shopper-style titles (Qwen3-8B), plain-language aliases per leaf
(Claude Sonnet), and public fashion catalog listings (Myntra, H&M) with LLM-voted labels.
These were used for training only, except for the aliases in
rr_aliases.json. - The image adapter was trained on embeddings of product photos from Amazon Reviews 2023 (Hou et al., McAuley Lab, UCSD) and Open Images crops. No images or review text are included in this repository.
Base model: jinaai/jina-clip-v1 (Apache-2.0).
Limitations
- US HTS only, with codes from the HTS revision above. Chapters 98 and 99 are excluded.
- Product titles that lack material or function details (e.g. "insulated bottle") can land in the wrong heading. The abstain flag catches many of these, but not all.
- Image-only queries are reliable at heading level and much less so at 10 digits.
- English only (non-English input is not translated at serve time).
- Accuracy drops sharply for codes with few past rulings.
Model tree for ruintheory/hts-classifier
Base model
jinaai/jina-clip-v1