INDEX.md

SAM 3 research experiments

One row per experiment, newest first. reconstructed = rebuilt from history for outputs that predate experiment logging.

date	experiment	question	status	conclusion
2026-10-15	vlm-tiers-gemma-4-31B-it-boxes	Can a small open VLM, shown one frame, name everything in it as primary / secondary / tertiary well enough for SAM 3 to turn the names into three tier masks -- and how often must it be asked again?	ok	Least unaccounted: google/gemma-4-31B-it (35% of the frame). Check sheets/ for whether the tiers make sense.
2026-10-15	vlm-tiers-Qwen3.5-9B-boxes	Can a small open VLM, shown one frame, name everything in it as primary / secondary / tertiary well enough for SAM 3 to turn the names into three tier masks -- and how often must it be asked again?	ok	Least unaccounted: Qwen/Qwen3.5-9B (29% of the frame). Check sheets/ for whether the tiers make sense.
2026-10-15	vlm-tiers-gemma-4-31B-it-v2	Can a small open VLM, shown one frame, name everything in it as primary / secondary / tertiary well enough for SAM 3 to turn the names into three tier masks -- and how often must it be asked again?	ok	Least unaccounted: google/gemma-4-31B-it (35% of the frame). Check sheets/ for whether the tiers make sense.
2026-10-15	unaccounted-audit-2026-10-14-vlm-tiers-gemma-4-31b-it	Of the frame no tier name covers, how much is mask edges, how much is genuinely unimportant, and how much is something a viewer would care about -- could a fourth "useless" tier take it?	ok	Of the unaccounted 35% of the frame: edges/scraps/'nothing' 30%, judged worth a tier 70%.
2026-10-14	stale-prompts-2026-10-12-vlm-tiers-qwen3-5-9b-2	If the VLM is asked once and SAM 3 keeps segmenting with those names, how fast does the answer go stale -- and when, on average, should the VLM be asked again?	ok	Median time to trigger within a shot: 1.0 s.
2026-10-14	vlm-tiers-InternVL3-5-38B-HF	Can a small open VLM, shown one frame, name everything in it as primary / secondary / tertiary well enough for SAM 3 to turn the names into three tier masks -- and how often must it be asked again?	ok	Least unaccounted: OpenGVLab/InternVL3_5-38B-HF (40% of the frame). Check sheets/ for whether the tiers make sense.
2026-10-14	vlm-tiers-gemma-4-31B-it	Can a small open VLM, shown one frame, name everything in it as primary / secondary / tertiary well enough for SAM 3 to turn the names into three tier masks -- and how often must it be asked again?	ok	Least unaccounted: google/gemma-4-31B-it (33% of the frame). Check sheets/ for whether the tiers make sense.
2026-10-12	vlm-tiers-Qwen3.5-9B	Can a small open VLM, shown one frame, name everything in it as primary / secondary / tertiary well enough for SAM 3 to turn the names into three tier masks -- and how often must it be asked again?	ok	Least unaccounted: Qwen/Qwen3.5-9B (35% of the frame). Check sheets/ for whether the tiers make sense.
2026-10-12	vlm-tiers-Qwen3.5-9B	Can a small open VLM, shown one frame, name everything in it as primary / secondary / tertiary well enough for SAM 3 to turn the names into three tier masks -- and how often must it be asked again?	failed	—
2026-10-12	everything-shots	How much of segment-everything's mask turnover on high-energy footage is shot cuts, how much is motion within a shot, and how much is the segmenter itself?	ok	Noise floor 82%; 1 s within a shot 40%; across a cut 7%; mean shot 8.1 s. Check the cut detector in cuts/.
2026-10-11	everything-stress	On high-energy footage (sports, gaming, live performance, music), how much does segment-everything cover, how many masks does it make, and how fast does the set of masks turn over from second to second?	ok	High-energy footage: 114 masks/frame covering 82%; 37% of masks persist across 1 s. Per-category rows and sheets/ show where it breaks.
2026-10-11	everything-ladder-bfloat16-points-per-side-64	How do segment-everything's settings trade speed against coverage and mask granularity, on diverse footage?	ok	{'dtype': 'bfloat16', 'points_per_side': 64, 'crops_n_layers': 0, 'pred_iou': 0.5, 'stability': 0.7}: 125.6 masks/frame covering 84% (overlap 2.29) in 4.33 s/frame on NVIDIA H200 NVL.
2026-10-11	everything-ladder-bfloat16	How do segment-everything's settings trade speed against coverage and mask granularity, on diverse footage?	ok	{'dtype': 'bfloat16', 'points_per_side': 32, 'crops_n_layers': 0, 'pred_iou': 0.5, 'stability': 0.7}: 77.8 masks/frame covering 81% (overlap 1.96) in 1.15 s/frame on NVIDIA H200 NVL.
2026-10-11	everything-ladder-bfloat16-points-per-side-16	How do segment-everything's settings trade speed against coverage and mask granularity, on diverse footage?	ok	{'dtype': 'bfloat16', 'points_per_side': 16, 'crops_n_layers': 0, 'pred_iou': 0.5, 'stability': 0.7}: 42.3 masks/frame covering 78% (overlap 1.62) in 0.34 s/frame on NVIDIA H200 NVL.
2026-10-11	everything-ladder-float32	How do segment-everything's settings trade speed against coverage and mask granularity, on diverse footage?	ok	{'dtype': 'float32', 'points_per_side': 32, 'crops_n_layers': 0, 'pred_iou': 0.5, 'stability': 0.7}: 77.3 masks/frame covering 82% (overlap 1.92) in 1.37 s/frame on NVIDIA H200 NVL.
2026-10-11	everything-ladder-bfloat16-crops-1	How do segment-everything's settings trade speed against coverage and mask granularity, on diverse footage?	failed	—
2026-10-11	everything-ladder-bfloat16-points-per-side-64	How do segment-everything's settings trade speed against coverage and mask granularity, on diverse footage?	ok	{'dtype': 'bfloat16', 'points_per_side': 64, 'crops_n_layers': 0, 'pred_iou': 0.5, 'stability': 0.7}: 89.2 masks/frame covering 72% (overlap 2.23) in 4.33 s/frame on NVIDIA H200 NVL.
2026-10-11	everything-ladder-float32	How do segment-everything's settings trade speed against coverage and mask granularity, on diverse footage?	ok	{'dtype': 'float32', 'points_per_side': 32, 'crops_n_layers': 0, 'pred_iou': 0.5, 'stability': 0.7}: 57.3 masks/frame covering 71% (overlap 1.97) in 1.35 s/frame on NVIDIA H200 NVL.
2026-10-11	everything-ladder-bfloat16	How do segment-everything's settings trade speed against coverage and mask granularity, on diverse footage?	ok	{'dtype': 'bfloat16', 'points_per_side': 32, 'crops_n_layers': 0, 'pred_iou': 0.5, 'stability': 0.7}: 56.5 masks/frame covering 71% (overlap 1.90) in 1.13 s/frame on NVIDIA H200 NVL.
2026-10-11	everything-ladder-bfloat16-points-per-side-16	How do segment-everything's settings trade speed against coverage and mask granularity, on diverse footage?	ok	{'dtype': 'bfloat16', 'points_per_side': 16, 'crops_n_layers': 0, 'pred_iou': 0.5, 'stability': 0.7}: 31.8 masks/frame covering 65% (overlap 1.58) in 0.33 s/frame on NVIDIA H200 NVL.
2026-10-06	hardware-profile-native-sam3.1	How long does each phase of SAM 3 video tracking take on this machine -- load, warm-up, setup, model time, rendering -- and how much does it vary between runs?	ok	NVIDIA A100-SXM4-40GB (native): load 16 s, warm-up 17 s, then per video setup 3.6 + model 68.1 + render 9.2 s.
2026-10-06	hardware-profile-transformers	How long does each phase of SAM 3 video tracking take on this machine -- load, warm-up, setup, model time, rendering -- and how much does it vary between runs?	ok	NVIDIA A100-SXM4-40GB (transformers): load 15 s, warm-up 27 s, then per video setup 3.5 + model 82.4 + render 8.3 s.
2026-10-05	hardware-profile-native-sam3.1	How long does each phase of SAM 3 video tracking take on this machine -- load, warm-up, setup, model time, rendering -- and how much does it vary between runs?	ok	NVIDIA H100 PCIe (native): load 15 s, warm-up 7 s, then per video setup 3.3 + model 17.0 + render 11.1 s.
2026-10-05	hardware-profile-transformers	How long does each phase of SAM 3 video tracking take on this machine -- load, warm-up, setup, model time, rendering -- and how much does it vary between runs?	ok	NVIDIA H100 PCIe (transformers): load 14 s, warm-up 12 s, then per video setup 3.6 + model 20.7 + render 9.9 s.
2026-10-04	native-classification	Without being told what to look for, what can SAM 3 find on its own -- which concepts from a fixed vocabulary it detects, how much of the frame "segment everything" covers, and how much of that it can name?	ok	Vocabulary sweep names 17.5 masks/frame (51% of the frame); segment-everything finds 57.3 (71%). See sheets/ for each frame.
2026-10-04	native-classification	Without being told what to look for, what can SAM 3 find on its own -- which concepts from a fixed vocabulary it detects, how much of the frame "segment everything" covers, and how much of that it can name?	failed	—
2026-10-04	hardware-profile-native-sam3.1	How long does each phase of SAM 3 video tracking take on this machine -- load, warm-up, setup, model time, rendering -- and how much does it vary between runs?	ok	NVIDIA L40S (native): load 17 s, warm-up 9 s, then per video setup 3.4 + model 27.6 + render 11.5 s.
2026-10-04	hardware-profile-transformers	How long does each phase of SAM 3 video tracking take on this machine -- load, warm-up, setup, model time, rendering -- and how much does it vary between runs?	ok	NVIDIA L40S (transformers): load 16 s, warm-up 15 s, then per video setup 3.6 + model 22.8 + render 9.8 s.
2026-10-04	hardware-profile-native-sam3.1	How long does each phase of SAM 3 video tracking take on this machine -- load, warm-up, setup, model time, rendering -- and how much does it vary between runs?	ok	NVIDIA H200 NVL (native): load 11 s, warm-up 8 s, then per video setup 1.5 + model 9.3 + render 2.9 s.
2026-10-04	hardware-profile-transformers	How long does each phase of SAM 3 video tracking take on this machine -- load, warm-up, setup, model time, rendering -- and how much does it vary between runs?	ok	NVIDIA H200 NVL (transformers): load 13 s, warm-up 8 s, then per video setup 1.2 + model 13.0 + render 2.7 s.
2026-10-04	hardware-profile-transformers	How long does each phase of SAM 3 video tracking take on this machine -- load, warm-up, setup, model time, rendering -- and how much does it vary between runs?	ok	Apple M3 Max (transformers): load 4 s, warm-up 71 s, then per video setup 1.6 + model 524.1 + render 2.7 s.
2026-10-04	tracking-backend-native-sam3.1-compile-pad-to-16	How fast is SAM 3 tracking with this backend configuration, compared with the transformers baseline on the same GPU?	ok	native / facebook/sam3.1 / attn=flash_attention_3 / compiled: inference 15.6 fps, 1.72x the transformers baseline. 2 ids.
2026-10-03	compile-shapes	What makes compiled SAM 3.1 recompile: video length, a leftover post-processing batch, or the way frames are loaded?	ok	Warm-up 219 s. Stalled passes: A same length, C leftover batch. Reasons in the job log (Recompiling ...).
2026-10-03	tracking-backend-native-sam3.1-compile-warm-up	How fast is SAM 3 tracking with this backend configuration, compared with the transformers baseline on the same GPU?	ok	native / facebook/sam3.1 / attn=flash_attention_3 / compiled: inference 7.5 fps, 0.83x the transformers baseline. 2 ids.
2026-10-03	tracking-backend-native-sam3.1-compile-warm-up	How fast is SAM 3 tracking with this backend configuration, compared with the transformers baseline on the same GPU?	ok	native / facebook/sam3.1 / attn=flash_attention_3 / compiled: inference 7.6 fps, 0.84x the transformers baseline. 2 ids.
2026-10-03	tracking-backend-native-sam3.1-compile	How fast is SAM 3 tracking with this backend configuration, compared with the transformers baseline on the same GPU?	ok	native / facebook/sam3.1 / attn=flash_attention_3 / compiled: inference 1.0 fps, 0.11x the transformers baseline. 2 ids.
2026-10-03	tracking-backend-native-sam3.1	How fast is SAM 3 tracking with this backend configuration, compared with the transformers baseline on the same GPU?	ok	native / facebook/sam3.1 / attn=flash_attention_3: inference 11.3 fps, 1.25x the transformers baseline. 2 ids.
2026-10-03	tracking-backend-native-sam3.1-sdpa	How fast is SAM 3 tracking with this backend configuration, compared with the transformers baseline on the same GPU?	ok	native / facebook/sam3.1 / attn=sdpa: inference 11.4 fps, 1.26x the transformers baseline. 2 ids.
2026-10-02	tracking-backend-native-sam3.1	How fast is SAM 3 tracking with this backend configuration, compared with the transformers baseline on the same GPU?	failed	—
2026-10-02	tracking-backend-native-sam3.1-sdpa	How fast is SAM 3 tracking with this backend configuration, compared with the transformers baseline on the same GPU?	failed	—
2026-10-02	tracking-backend-backend-native-compile	How fast is SAM 3 tracking with this backend configuration, compared with the transformers baseline on the same GPU?	ok	native / facebook/sam3 / attn=sdpa / compiled: inference 12.4 fps, 1.37x the transformers baseline. 2 ids.
2026-10-02	tracking-backend-backend-native	How fast is SAM 3 tracking with this backend configuration, compared with the transformers baseline on the same GPU?	ok	native / facebook/sam3 / attn=sdpa: inference 8.3 fps, 0.91x the transformers baseline. 2 ids.
2026-10-02	tracking-vs-frames	On a full clip, how much does SAM 3's tracker cost compared with segmenting every frame independently, and is frame-by-frame segmentation stable enough without it?	ok	Inference only: per-frame 14.6 fps vs tracker 9.0 fps (1.6x); tracker setup 3.6 s, rendering ~9.8 s. Per-frame + IoU matching produced 17 ids (tracker 2) and the count left its usual value on 12/200 frames. Compare the two videos in runs/.
2026-10-02	tracking-vs-frames	On a full clip, how much does SAM 3's tracker cost compared with segmenting every frame independently, and is frame-by-frame segmentation stable enough without it?	ok	Inference only: per-frame 14.7 fps vs tracker 9.0 fps (1.6x); tracker setup 3.6 s, rendering ~9.9 s. Per-frame + IoU matching produced 17 ids (tracker 2) and the count left its usual value on 12/200 frames. Compare the two videos in runs/.
2026-10-02	tracking-vs-frames	On a full clip, how much does SAM 3's tracker cost compared with segmenting every frame independently, and is frame-by-frame segmentation stable enough without it?	ok	Inference only: per-frame 14.6 fps vs tracker 9.1 fps (1.6x); tracker setup 3.6 s, rendering ~9.9 s. Per-frame + IoU matching produced 17 ids (tracker 2) and the count left its usual value on 12/200 frames. Compare the two videos in runs/.
2026-10-02	tracking-vs-frames	On a full clip, how much does SAM 3's tracker cost compared with segmenting every frame independently, and is frame-by-frame segmentation stable enough without it?	ok	Inference only: per-frame 14.4 fps vs tracker 8.8 fps (1.6x); tracker setup 3.5 s, rendering ~9.8 s. Per-frame + IoU matching produced 17 ids (tracker 2) and the count left its usual value on 12/200 frames. Compare the two videos in runs/.
2026-10-02	tracking-vs-frames	On a full clip, how much does SAM 3's tracker cost compared with segmenting every frame independently, and what does tracking buy?	ok	Frame-by-frame segmentation ran at 16.1 fps vs the tracker's 5.0 fps (3.2x), i.e. 1.87x vs 6.05x the clip's length. Without tracking, the object count changed on 12 of 200 frames.
2026-10-02	backbone-head-split	How is per-frame cost split between the vision backbone and the per-prompt head?	ok	The backbone is 49% of per-frame cost. Computing it once per frame and reusing it for every prompt makes each extra object cost ~0.04 s instead of ~0.07 s (1.7x faster for 8 prompts).
2026-09-29	backbone-head-split	How is per-frame cost split between the vision backbone and the per-prompt head?	ok	The backbone is 48% of per-frame cost. Computing it once per frame and reusing it for every prompt makes each extra object cost ~0.04 s instead of ~0.07 s (1.7x faster for 8 prompts).
2026-09-24	resolution-ladder	Does input resolution change SAM 3's tracking speed, or its ability to find the object?	ok	Speed ranged 0.55-0.61 fps across a 256x pixel range. 'bed' was found in every frame at every size, down to 60x34.
2026-09-24	batch-size-sweep	Does batching frames through the image model (per-frame segmentation, no tracking) raise throughput?	ok	Best was batch 2 at 0.95 fps, 1.06x batch 1. Batching gives no real gain here: one frame already saturates the GPU.
2026-09-24	backbone-head-split	How is per-frame cost split between the vision backbone and the per-prompt head?	ok	The backbone is 86% of per-frame cost. Computing it once per frame and reusing it for every prompt makes each extra object cost ~0.11 s instead of ~1.04 s (4.5x faster for 8 prompts).
2026-09-21	hardware-m3-vs-t4	Is a Colab T4 faster than the M3 Max for SAM 3 video tracking?	reconstructed	No: the T4 is ~1.8x slower than the laptop, and fp16 barely helped -- the T4 is limited by memory bandwidth. L4/A100 are the first GPUs where bf16 and FlashAttention-2 engage.
2026-09-15	harness-regression-tests	After each refactor, do image and video runs still give the same detections, with correct timings?	reconstructed	Detections stayed identical across every refactor. Per-frame timing was wrong until it moved to forward hooks, because the tracker buffers its first 15 frames.
2026-09-15	resolution-ladder	Does input resolution change SAM 3's speed, or its ability to find the object?	reconstructed	No: speed is flat at 0.60-0.63 fps because SAM 3 resamples every input to 1008x1008, and the bed was still tracked 40/40 frames at 60x34.
2026-09-12	legend-autofit	Does the diagnostic legend stay readable and inside the image as image size shrinks?	reconstructed	Originally no -- at 73x110 the panel covered the image. Fixed 9/13 by shrinking the font to fit, then dropping rows.
2026-09-12	first-image-tests	Does SAM 3 (transformers port, M3 Max) find every instance of a text prompt in a still image?	reconstructed	Yes: all 4 truck wheels were found, including the 2 partly hidden ones, at ~1.1 s per image.
2026-09-12	first-tracking-test	Can SAM 3 video PCS detect and track a text prompt through a full clip, and how fast on the M3 Max?	reconstructed	Yes, identity held for all 200 frames, but at 0.52 fps -- about 58x slower than real time.

Runs outside any experiment: experiments/adhoc