pasma / EVALUATION_DETAILS.md

Publish Pasma v1.0 candidates, model card and complete benchmark evidence

028c4b5 verified 2 days ago

10 kB

# Pasma v1.0 model card

**Card date:** 2026-10-21. **Status:** v1.0 candidate release. Weights are hosted in this repository under Apache-2.0 with retained upstream notices. See the deployment limitations and provenance ledger below.

## Model and intended use

Pasma classifies one main garment/accessory in a home photograph for wardrobe entry, catalogue assistance and user-correctable suggestions. Small and Nano use `edgenext_small.usi_in1k`; Pico uses `convnext_pico.d1_in1k`. Inputs are 320² or 256² RGB; outputs are 20-way type softmax, six category probability sums, and 21 independent visual-material sigmoids. It needs no background removal model.

Training uses a shared backbone with type, auxiliary coarse and material heads. The exported graph discards the independent coarse prediction and computes `prob_coarse = prob_type @ H`. There are no weather/style heads in these artifacts. This release follows the later Pasma plan's v3 schema, which differs from the original four-head development specification; the documentation does not imply compliance with that older architecture.

| Variant | Run | Parameters (M) | Best epoch | Local eval T | Export T |
|---|---|---:|---:|---:|---:|
| Pasma-Small-320 | `r4_edgenext_320` | 5.296 | 27 | 0.8846 | 0.8877 |
| Pasma-Nano-256 | `r3_edgenext_s` | 5.296 | 30 | 0.8901 | 0.8912 |
| Pasma-Pico-256 | `r3_convnext_pico` | 8.557 | 39 | 0.9078 | 0.9071 |

Parameter counts are measured for the fitted classification models, not the original ImageNet classifiers (often quoted as 5.6M/9.0M). Export and local-evaluation temperatures come from different recorded fits and must not be silently interchanged. Softmax temperature preserves type ranking but changes calibration and potentially aggregated coarse ranking.

## Training and data

The released students' saved configs use real-only training with a fine-tuned `vit_so400m_patch16_siglip_256.v2_webli` teacher (`r2_so400m_teacher`). The broader research pipeline includes synthetic data and real/synthetic teacher training; the students are not documented as directly trained on a synthetic/real mixture or an ensemble. See the [technical report](docs/PASMA_TECHNICAL_REPORT.md) for the implemented loss.

Common student settings: seed 42, 40 epochs, batch 64, 12,000 sampled examples per epoch, AdamW head LR 5e-4, backbone LR multiplier 0.5, weight decay 0.05, one warm-up epoch followed by cosine decay, EMA 0.999, dropout 0.2, drop-path 0.1, label smoothing 0.1, online type distillation alpha 0.9 at T=2, and auxiliary coarse/material weights 0.3/0.3. The selected checkpoint maximizes validation Top-1 + macro F1. Unknown/low-confidence materials are excluded from material loss.

The fixed v3 manifest is `data/splits/v3_splits.csv`; ordinary train/validation/test and benchmark-development/test assignments are stored there. No random split is made during evaluation. Embedding/dHash screening reduces known overlap; it is not proof that all physical-item duplicates are absent. The [taxonomy/protocol](docs/TAXONOMY_AND_BENCHMARK.md) records its scope.

Source-derived type labels and curated expected-material sets have no supplied complete double-annotation/adjudication record. Material targets represent plausible visual labels, not laboratory fibre measurements. Image rights, participant consent, generator/pseudo-label provider provenance and redistribution permissions require a separate release ledger. Private source photos are excluded from the evidence snapshot.

## Evaluation

| Model | Input | ONNX FP16 (MiB) | test_real Top-1 [95% CI] | bench_test Top-1 [95% CI] | Combined Top-1 [95% CI] | Combined macro F1 |
|---|---|---:|---|---|---|---:|
| Pasma-Small-320 | 320² | 10.32 | 90.75% [88.8, 92.6] | 83.53% [78.7, 87.9] | 89.12% [87.4, 90.9] | 87.01% |
| Pasma-Nano-256 | 256² | 10.32 | 90.05% [87.9, 92.0] | 80.72% [75.5, 85.5] | 87.94% [86.0, 89.8] | 85.44% |
| Pasma-Pico-256 | 256² | 16.40 | 89.11% [87.0, 91.2] | 81.93% [77.1, 86.8] | 87.49% [85.5, 89.4] | 84.84% |

Type Top-3 on test_real / bench_test / combined: Small **98.01 / 93.98 / 97.10%**; Nano **98.13 / 93.17 / 97.01%**; Pico **97.78 / 93.57 / 96.83%**. The teacher reaches **93.93%** combined Top-1 (95% CI 92.6-95.2); Small reaches 94.9% of that accuracy, a descriptive ratio rather than a statistical equivalence claim.

The type CIs use 2,000 image-level bootstrap resamples, seed 0. They do not model group dependence, label noise, training seed variation or repeated model selection. Local FP32 evaluation uses MPS, no TTA and `exif=false`; deployment requires EXIF correction. The source Modal BF16 results differ by a few borderline images and are not mixed into these tables.

| Model | Combined coarse accuracy | ECE test_real / bench_test | Material mAP on bench_test [95% CI] | Material supported macro F1 / micro F1 |
|---|---:|---:|---|---:|
| Small-320 | 95.19% | 0.0466 / 0.1016 | 0.664 [0.630, 0.715] | 0.624 / 0.690 |
| Nano-256 | 94.56% | 0.0411 / 0.1059 | 0.654 [0.624, 0.703] | 0.646 / 0.699 |
| Pico-256 | 93.83% | 0.0443 / 0.1059 | 0.638 [0.610, 0.685] | 0.600 / 0.659 |
Material thresholds are fitted on `bench_dev`, then fixed for `bench_test`. Only 17/21 classes have positive test support; mAP and supported macro F1 exclude unsupported classes. The exact metrics, 1,000-resample material CIs, all-class F1 and per-class results are in the [evidence snapshot](docs/benchmarks/pasma_v1_evidence.json).
### Per-type recall / F1
Calculated from the saved confusion matrices on `real_all_heldout` (n=1,103); support is shared across variants. These are point estimates, not per-class confidence intervals.
| Type | Support | Small R / F1 | Nano R / F1 | Pico R / F1 |
|---|---:|---:|---:|---:|
| T_SHIRT | 67 | 0.910 / 0.871 | 0.910 / 0.853 | 0.955 / 0.859 |
| SHIRT | 38 | 0.895 / 0.829 | 0.842 / 0.800 | 0.842 / 0.800 |
| BLOUSE | 23 | 0.478 / 0.564 | 0.522 / 0.558 | 0.435 / 0.526 |
| SWEATER | 53 | 0.925 / 0.907 | 0.925 / 0.891 | 0.925 / 0.916 |
| HOODIE | 24 | 0.917 / 0.898 | 0.792 / 0.826 | 0.833 / 0.870 |
| PANTS | 89 | 0.944 / 0.939 | 0.921 / 0.911 | 0.921 / 0.911 |
| JEANS | 21 | 0.857 / 0.923 | 0.810 / 0.872 | 0.714 / 0.769 |
| SHORTS | 28 | 0.893 / 0.893 | 0.786 / 0.830 | 0.857 / 0.889 |
| SKIRT | 59 | 0.949 / 0.949 | 0.949 / 0.926 | 0.932 / 0.948 |
| JACKET | 69 | 0.797 / 0.859 | 0.754 / 0.812 | 0.739 / 0.816 |
| COAT | 117 | 0.966 / 0.930 | 0.940 / 0.905 | 0.949 / 0.906 |
| VEST | 25 | 0.720 / 0.800 | 0.680 / 0.773 | 0.640 / 0.744 |
| DRESS | 67 | 0.910 / 0.897 | 0.896 / 0.916 | 0.851 / 0.884 |
| SHOES | 54 | 0.852 / 0.807 | 0.907 / 0.852 | 0.852 / 0.807 |
| BOOTS | 48 | 0.896 / 0.915 | 0.875 / 0.913 | 0.854 / 0.882 |
| SNEAKERS | 46 | 0.913 / 0.875 | 0.913 / 0.884 | 0.935 / 0.915 |
| SANDALS | 58 | 0.776 / 0.811 | 0.793 / 0.844 | 0.776 / 0.811 |
| ACCESSORY | 154 | 0.942 / 0.942 | 0.955 / 0.942 | 0.968 / 0.931 |
| HAT | 35 | 0.971 / 0.986 | 0.971 / 0.971 | 0.886 / 0.912 |
| SCARF | 28 | 0.750 / 0.808 | 0.750 / 0.808 | 0.857 / 0.873 |

![Small-320 confusion matrix on combined held-out photos](docs/assets/pasma/confusion_small_real_all.png)

## Deployment and numerical behavior

ONNX uses opset 17, batch 1, float32 input/output even for FP16/INT8 internal weights. CoreML uses FP16 MLProgram with an iOS 17 deployment target. FP32 ONNX matches the reference in the saved type-parity test; FP16 and INT8 do not all meet the original max probability error requirement. See [deployment](docs/DEPLOYMENT.md) for exact measurements and [release gates](docs/DEPLOYMENT.md) for pending work. M2 CPU+NE p50 is 5.85/4.21/2.27 ms for Small/Nano/Pico; phone latency, peak activation RAM and browser/Android parity are unmeasured.

## Limitations and ethical considerations

- BLOUSE is weak: Small recall 11/23 (47.8%) and Top-3 recall 21/23 (91.3%). An exploratory +2.3 logit adjustment is excluded from released graphs/headline metrics; it trades SHIRT recall for BLOUSE recall and needs fresh validation.
- Use Top-3 suggestions and user confirmation. A confidence threshold is a product policy to validate on deployment data; it is not a universal reliability guarantee. Calibration worsens on `bench_test`.
- Coarse probabilities are consistent marginals, but independent type/category argmax choices can disagree. Map the chosen type to its category for a consistent displayed pair.
- Multiple garments, tiny/cropped items, unusual silhouettes, backgrounds and ambiguous taxonomy boundaries can cause failures. There is no explicit unknown/out-of-distribution class or measured rejection benchmark.
- No demographic, geographic, device, lighting or accessibility fairness study is supplied. Performance cannot be assumed uniform across these groups.
- Do not use visual material outputs to certify fibre content, allergy safety, care instructions, price, authenticity or provenance. Appearance can be shared by different fibres and blends.
- Offline inference can keep images on the device; application telemetry/upload behavior is outside this model's control. Cloud benchmark calls send images to external providers and are separate from local inference.
- The private dataset prevents complete independent reproduction from a source checkout. Published aggregate metrics allow inspection, not verification of unavailable labels/photos.

## License, maintenance and citation

Project code is Apache-2.0. Intended weight release is Apache-2.0 subject to upstream retained notices and completed provenance checks; see [license ledger](docs/PROVENANCE_AND_LICENSES.md). No benchmark-only third-party weights or segmentation weights are included. This card does not grant rights to source datasets.

Report issues through the repository issue tracker with runtime, model SHA-256 and a non-sensitive reproduction. Cite this card as *Pasma v1.0: Compact Clothing Classification for Home Wardrobe Photographs*, technical report, 2026; a canonical URL/DOI and named author list will be added at publication. No paper acceptance or arXiv submission is claimed.


	# Pasma v1.0 model card

	Card date: 2026-10-21. Status: v1.0 candidate release. Weights are hosted in this repository under Apache-2.0 with retained upstream notices. See the deployment limitations and provenance ledger below.

	## Model and intended use

	Pasma classifies one main garment/accessory in a home photograph for wardrobe entry, catalogue assistance and user-correctable suggestions. Small and Nano use `edgenext_small.usi_in1k`; Pico uses `convnext_pico.d1_in1k`. Inputs are 320² or 256² RGB; outputs are 20-way type softmax, six category probability sums, and 21 independent visual-material sigmoids. It needs no background removal model.

	Training uses a shared backbone with type, auxiliary coarse and material heads. The exported graph discards the independent coarse prediction and computes `prob_coarse = prob_type @ H`. There are no weather/style heads in these artifacts. This release follows the later Pasma plan's v3 schema, which differs from the original four-head development specification; the documentation does not imply compliance with that older architecture.

	\| Variant \| Run \| Parameters (M) \| Best epoch \| Local eval T \| Export T \|
	\|---\|---\|---:\|---:\|---:\|---:\|
	\| Pasma-Small-320 \| `r4_edgenext_320` \| 5.296 \| 27 \| 0.8846 \| 0.8877 \|
	\| Pasma-Nano-256 \| `r3_edgenext_s` \| 5.296 \| 30 \| 0.8901 \| 0.8912 \|
	\| Pasma-Pico-256 \| `r3_convnext_pico` \| 8.557 \| 39 \| 0.9078 \| 0.9071 \|

	Parameter counts are measured for the fitted classification models, not the original ImageNet classifiers (often quoted as 5.6M/9.0M). Export and local-evaluation temperatures come from different recorded fits and must not be silently interchanged. Softmax temperature preserves type ranking but changes calibration and potentially aggregated coarse ranking.

	## Training and data

	The released students' saved configs use real-only training with a fine-tuned `vit_so400m_patch16_siglip_256.v2_webli` teacher (`r2_so400m_teacher`). The broader research pipeline includes synthetic data and real/synthetic teacher training; the students are not documented as directly trained on a synthetic/real mixture or an ensemble. See the [technical report](docs/PASMA_TECHNICAL_REPORT.md) for the implemented loss.

	Common student settings: seed 42, 40 epochs, batch 64, 12,000 sampled examples per epoch, AdamW head LR 5e-4, backbone LR multiplier 0.5, weight decay 0.05, one warm-up epoch followed by cosine decay, EMA 0.999, dropout 0.2, drop-path 0.1, label smoothing 0.1, online type distillation alpha 0.9 at T=2, and auxiliary coarse/material weights 0.3/0.3. The selected checkpoint maximizes validation Top-1 + macro F1. Unknown/low-confidence materials are excluded from material loss.

	The fixed v3 manifest is `data/splits/v3_splits.csv`; ordinary train/validation/test and benchmark-development/test assignments are stored there. No random split is made during evaluation. Embedding/dHash screening reduces known overlap; it is not proof that all physical-item duplicates are absent. The [taxonomy/protocol](docs/TAXONOMY_AND_BENCHMARK.md) records its scope.

	Source-derived type labels and curated expected-material sets have no supplied complete double-annotation/adjudication record. Material targets represent plausible visual labels, not laboratory fibre measurements. Image rights, participant consent, generator/pseudo-label provider provenance and redistribution permissions require a separate release ledger. Private source photos are excluded from the evidence snapshot.

	## Evaluation

	\| Model \| Input \| ONNX FP16 (MiB) \| test_real Top-1 [95% CI] \| bench_test Top-1 [95% CI] \| Combined Top-1 [95% CI] \| Combined macro F1 \|
	\|---\|---\|---:\|---\|---\|---\|---:\|
	\| Pasma-Small-320 \| 320² \| 10.32 \| 90.75% [88.8, 92.6] \| 83.53% [78.7, 87.9] \| 89.12% [87.4, 90.9] \| 87.01% \|
	\| Pasma-Nano-256 \| 256² \| 10.32 \| 90.05% [87.9, 92.0] \| 80.72% [75.5, 85.5] \| 87.94% [86.0, 89.8] \| 85.44% \|
	\| Pasma-Pico-256 \| 256² \| 16.40 \| 89.11% [87.0, 91.2] \| 81.93% [77.1, 86.8] \| 87.49% [85.5, 89.4] \| 84.84% \|

	Type Top-3 on test_real / bench_test / combined: Small 98.01 / 93.98 / 97.10%; Nano 98.13 / 93.17 / 97.01%; Pico 97.78 / 93.57 / 96.83%. The teacher reaches 93.93% combined Top-1 (95% CI 92.6-95.2); Small reaches 94.9% of that accuracy, a descriptive ratio rather than a statistical equivalence claim.

	The type CIs use 2,000 image-level bootstrap resamples, seed 0. They do not model group dependence, label noise, training seed variation or repeated model selection. Local FP32 evaluation uses MPS, no TTA and `exif=false`; deployment requires EXIF correction. The source Modal BF16 results differ by a few borderline images and are not mixed into these tables.

	\| Model \| Combined coarse accuracy \| ECE test_real / bench_test \| Material mAP on bench_test [95% CI] \| Material supported macro F1 / micro F1 \|
	\|---\|---:\|---:\|---\|---:\|
	\| Small-320 \| 95.19% \| 0.0466 / 0.1016 \| 0.664 [0.630, 0.715] \| 0.624 / 0.690 \|
	\| Nano-256 \| 94.56% \| 0.0411 / 0.1059 \| 0.654 [0.624, 0.703] \| 0.646 / 0.699 \|
	\| Pico-256 \| 93.83% \| 0.0443 / 0.1059 \| 0.638 [0.610, 0.685] \| 0.600 / 0.659 \|

	Material thresholds are fitted on `bench_dev`, then fixed for `bench_test`. Only 17/21 classes have positive test support; mAP and supported macro F1 exclude unsupported classes. The exact metrics, 1,000-resample material CIs, all-class F1 and per-class results are in the [evidence snapshot](docs/benchmarks/pasma_v1_evidence.json).

	### Per-type recall / F1

	Calculated from the saved confusion matrices on `real_all_heldout` (n=1,103); support is shared across variants. These are point estimates, not per-class confidence intervals.

	\| Type \| Support \| Small R / F1 \| Nano R / F1 \| Pico R / F1 \|
	\|---\|---:\|---:\|---:\|---:\|
	\| T_SHIRT \| 67 \| 0.910 / 0.871 \| 0.910 / 0.853 \| 0.955 / 0.859 \|
	\| SHIRT \| 38 \| 0.895 / 0.829 \| 0.842 / 0.800 \| 0.842 / 0.800 \|
	\| BLOUSE \| 23 \| 0.478 / 0.564 \| 0.522 / 0.558 \| 0.435 / 0.526 \|
	\| SWEATER \| 53 \| 0.925 / 0.907 \| 0.925 / 0.891 \| 0.925 / 0.916 \|
	\| HOODIE \| 24 \| 0.917 / 0.898 \| 0.792 / 0.826 \| 0.833 / 0.870 \|
	\| PANTS \| 89 \| 0.944 / 0.939 \| 0.921 / 0.911 \| 0.921 / 0.911 \|
	\| JEANS \| 21 \| 0.857 / 0.923 \| 0.810 / 0.872 \| 0.714 / 0.769 \|
	\| SHORTS \| 28 \| 0.893 / 0.893 \| 0.786 / 0.830 \| 0.857 / 0.889 \|
	\| SKIRT \| 59 \| 0.949 / 0.949 \| 0.949 / 0.926 \| 0.932 / 0.948 \|
	\| JACKET \| 69 \| 0.797 / 0.859 \| 0.754 / 0.812 \| 0.739 / 0.816 \|
	\| COAT \| 117 \| 0.966 / 0.930 \| 0.940 / 0.905 \| 0.949 / 0.906 \|
	\| VEST \| 25 \| 0.720 / 0.800 \| 0.680 / 0.773 \| 0.640 / 0.744 \|
	\| DRESS \| 67 \| 0.910 / 0.897 \| 0.896 / 0.916 \| 0.851 / 0.884 \|
	\| SHOES \| 54 \| 0.852 / 0.807 \| 0.907 / 0.852 \| 0.852 / 0.807 \|
	\| BOOTS \| 48 \| 0.896 / 0.915 \| 0.875 / 0.913 \| 0.854 / 0.882 \|
	\| SNEAKERS \| 46 \| 0.913 / 0.875 \| 0.913 / 0.884 \| 0.935 / 0.915 \|
	\| SANDALS \| 58 \| 0.776 / 0.811 \| 0.793 / 0.844 \| 0.776 / 0.811 \|
	\| ACCESSORY \| 154 \| 0.942 / 0.942 \| 0.955 / 0.942 \| 0.968 / 0.931 \|
	\| HAT \| 35 \| 0.971 / 0.986 \| 0.971 / 0.971 \| 0.886 / 0.912 \|
	\| SCARF \| 28 \| 0.750 / 0.808 \| 0.750 / 0.808 \| 0.857 / 0.873 \|

	![Small-320 confusion matrix on combined held-out photos](docs/assets/pasma/confusion_small_real_all.png)

	## Deployment and numerical behavior

	ONNX uses opset 17, batch 1, float32 input/output even for FP16/INT8 internal weights. CoreML uses FP16 MLProgram with an iOS 17 deployment target. FP32 ONNX matches the reference in the saved type-parity test; FP16 and INT8 do not all meet the original max probability error requirement. See [deployment](docs/DEPLOYMENT.md) for exact measurements and [release gates](docs/DEPLOYMENT.md) for pending work. M2 CPU+NE p50 is 5.85/4.21/2.27 ms for Small/Nano/Pico; phone latency, peak activation RAM and browser/Android parity are unmeasured.

	## Limitations and ethical considerations

	- BLOUSE is weak: Small recall 11/23 (47.8%) and Top-3 recall 21/23 (91.3%). An exploratory +2.3 logit adjustment is excluded from released graphs/headline metrics; it trades SHIRT recall for BLOUSE recall and needs fresh validation.
	- Use Top-3 suggestions and user confirmation. A confidence threshold is a product policy to validate on deployment data; it is not a universal reliability guarantee. Calibration worsens on `bench_test`.
	- Coarse probabilities are consistent marginals, but independent type/category argmax choices can disagree. Map the chosen type to its category for a consistent displayed pair.
	- Multiple garments, tiny/cropped items, unusual silhouettes, backgrounds and ambiguous taxonomy boundaries can cause failures. There is no explicit unknown/out-of-distribution class or measured rejection benchmark.
	- No demographic, geographic, device, lighting or accessibility fairness study is supplied. Performance cannot be assumed uniform across these groups.
	- Do not use visual material outputs to certify fibre content, allergy safety, care instructions, price, authenticity or provenance. Appearance can be shared by different fibres and blends.
	- Offline inference can keep images on the device; application telemetry/upload behavior is outside this model's control. Cloud benchmark calls send images to external providers and are separate from local inference.
	- The private dataset prevents complete independent reproduction from a source checkout. Published aggregate metrics allow inspection, not verification of unavailable labels/photos.

	## License, maintenance and citation

	Project code is Apache-2.0. Intended weight release is Apache-2.0 subject to upstream retained notices and completed provenance checks; see [license ledger](docs/PROVENANCE_AND_LICENSES.md). No benchmark-only third-party weights or segmentation weights are included. This card does not grant rights to source datasets.

	Report issues through the repository issue tracker with runtime, model SHA-256 and a non-sensitive reproduction. Cite this card as Pasma v1.0: Compact Clothing Classification for Home Wardrobe Photographs, technical report, 2026; a canonical URL/DOI and named author list will be added at publication. No paper acceptance or arXiv submission is claimed.