diff --git a/.gitattributes b/.gitattributes index a6344aac8c09253b3b630fb776ae94478aa0275b..f1ca65ce6a5b0774360fde1ec05157d9f937aaa2 100644 --- a/.gitattributes +++ b/.gitattributes @@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text *.zip filter=lfs diff=lfs merge=lfs -text *.zst filter=lfs diff=lfs merge=lfs -text *tfevents* filter=lfs diff=lfs merge=lfs -text +tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..c9cc77999905a89e9fec9593091b68f98f33a235 --- /dev/null +++ b/README.md @@ -0,0 +1,88 @@ +--- +license: other +license_name: mixed-non-commercial-research +base_model: Qwen/Qwen3-8B +library_name: peft +tags: +- 3d-scene-layout +- indoor-scenes +- furniture-placement +- qwen3 +- lora +--- + +# FastFill (v4, base): object-conditioned indoor room layout + +FastFill lays out the furniture of one room in a **single forward pass**. Given: +- the room's floor polygon, doors and windows, and fixed objects; +- the list of objects to place, with categories and optional descriptions; +- optional constraints; + +it predicts, for **every** object: +- a size `[depth, width, height]` in metres; +- a bottom-centre position `[x, y, z]` in metres; +- a yaw: the direction the object's front faces, counter-clockwise from +x. + +This repository holds what you need beyond the base model: +- the LoRA adapter on Qwen3-8B; +- the geometry decoder and heads; +- the tokenizer; +- the exact FastFill code the model was trained with. + +**Download `Qwen/Qwen3-8B` and run it.** Nothing else is needed. + +## Use + +```bash +pip install torch # the build for your CUDA / platform +pip install -r requirements.txt +python predict.py example_condition.json # fetches Qwen/Qwen3-8B from the Hub +python predict.py example_condition.json --qwen /path/to/Qwen3-8B # or a local copy +``` + +In Python: + +```python +import json, predict +model, tokenizer, max_length = predict.load(qwen="Qwen/Qwen3-8B", device="cuda") +condition = json.load(open("example_condition.json")) +layout = predict.predict_layout(model, tokenizer, condition, max_length=max_length, device="cuda", + grid_decode="spread") +``` + +**Hardware:** the bf16 base model needs about 16 GB of GPU memory. One room takes well under a second on a GH200. + +## Input and output + +- **Input:** a `fastfill.v2` condition. See `example_condition.json` and `code/fastfill/v2/schema.py`. + - `room`: `floor_polygon_xy_m`, `floor_z_m`, `height_m`, `openings` (doors and windows), `fixed_objects`. + - `objects`: `id`, `category`, optional `description` and `support_parent`. + - `constraints`: `against_wall`, `near`, `faces`, `between`, ... +- **Output:** `{"schema_version": "fastfill.v2", "objects": [{"id", "target_size_local_m", "bottom_center_m", "yaw_rad"}, ...]}`, one entry per requested object. + +## Model + +| | | +|---|---| +| Base model | Qwen3-8B, frozen, bf16 | +| Adapter | LoRA rank 8, alpha 16, on q/k/v/o projections | +| Decoder and heads | 2-layer object decoder (dim 128, 4 heads)
size head (log-space)
yaw: 12 bins plus a residual
position: 16×16 grid cell plus a residual | +| Training | `main4-cell05-v4fixed-e5-base`: 6,552 steps (5 epochs)
effective batch 24, lr 1e-4 with cosine schedule
max length 8,192 tokens, up to 256 objects per room | +| Data | FastFill v4 layout data (v2-fixed build)
indoor rooms from about 20 public sources | +| Code | `code/fastfill`: training commit `26939bc`
the model, batch, evaluate, io and schema modules are byte-identical to it | + +`checkpoint/training_manifest.json` records the training run (data hashes, full configuration). + +## Results (test300: 300 held-out rooms, stratified by source) + +| Decoding | Position error (m) | Size error (log) | Yaw error (rad) | Out of bounds | Blocks a door | Collisions per room | Constraints met | Clean rooms | +|---|---|---|---|---|---|---|---|---| +| spread | 3.11 | **0.321** | 0.593 | 10.0% | 11.7% | 3.65 | 37.7% | 11.1% | + +**Read the raw output as a draft. It is not a guaranteed-valid layout.** Sizes are the model's strongest output, and they are what our downstream workflow uses. Positions often collide, block doors or break constraints. Pair the model with a geometric checker or repair step before using a layout in simulation. + +## Limitations + +- **One room at a time.** The model reads furniture boxes only, not meshes or appearance. +- **No validity guarantee.** Collisions, out-of-bounds objects and blocked doors occur, as shown in the table above. +- **Licence.** The training data mixes sources released for non-commercial research; use this model under the same terms. The base model, Qwen3-8B, is Apache-2.0. diff --git a/checkpoint/backbone/README.md b/checkpoint/backbone/README.md new file mode 100644 index 0000000000000000000000000000000000000000..9612927f991d544fc60edc337e1adc821de4b442 --- /dev/null +++ b/checkpoint/backbone/README.md @@ -0,0 +1,206 @@ +--- +base_model: /lus/lfs1aip2/projects/u6uv/shanliantian.u6uv/fastfill-v4/models/Qwen3-8B +library_name: peft +tags: +- base_model:adapter:/lus/lfs1aip2/projects/u6uv/shanliantian.u6uv/fastfill-v4/models/Qwen3-8B +- lora +- transformers +--- + +# Model Card for Model ID + + + + + +## Model Details + +### Model Description + + + + + +- **Developed by:** [More Information Needed] +- **Funded by [optional]:** [More Information Needed] +- **Shared by [optional]:** [More Information Needed] +- **Model type:** [More Information Needed] +- **Language(s) (NLP):** [More Information Needed] +- **License:** [More Information Needed] +- **Finetuned from model [optional]:** [More Information Needed] + +### Model Sources [optional] + + + +- **Repository:** [More Information Needed] +- **Paper [optional]:** [More Information Needed] +- **Demo [optional]:** [More Information Needed] + +## Uses + + + +### Direct Use + + + +[More Information Needed] + +### Downstream Use [optional] + + + +[More Information Needed] + +### Out-of-Scope Use + + + +[More Information Needed] + +## Bias, Risks, and Limitations + + + +[More Information Needed] + +### Recommendations + + + +Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations. + +## How to Get Started with the Model + +Use the code below to get started with the model. + +[More Information Needed] + +## Training Details + +### Training Data + + + +[More Information Needed] + +### Training Procedure + + + +#### Preprocessing [optional] + +[More Information Needed] + + +#### Training Hyperparameters + +- **Training regime:** [More Information Needed] + +#### Speeds, Sizes, Times [optional] + + + +[More Information Needed] + +## Evaluation + + + +### Testing Data, Factors & Metrics + +#### Testing Data + + + +[More Information Needed] + +#### Factors + + + +[More Information Needed] + +#### Metrics + + + +[More Information Needed] + +### Results + +[More Information Needed] + +#### Summary + + + +## Model Examination [optional] + + + +[More Information Needed] + +## Environmental Impact + + + +Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). + +- **Hardware Type:** [More Information Needed] +- **Hours used:** [More Information Needed] +- **Cloud Provider:** [More Information Needed] +- **Compute Region:** [More Information Needed] +- **Carbon Emitted:** [More Information Needed] + +## Technical Specifications [optional] + +### Model Architecture and Objective + +[More Information Needed] + +### Compute Infrastructure + +[More Information Needed] + +#### Hardware + +[More Information Needed] + +#### Software + +[More Information Needed] + +## Citation [optional] + + + +**BibTeX:** + +[More Information Needed] + +**APA:** + +[More Information Needed] + +## Glossary [optional] + + + +[More Information Needed] + +## More Information [optional] + +[More Information Needed] + +## Model Card Authors [optional] + +[More Information Needed] + +## Model Card Contact + +[More Information Needed] +### Framework versions + +- PEFT 0.21.0 \ No newline at end of file diff --git a/checkpoint/backbone/adapter_config.json b/checkpoint/backbone/adapter_config.json new file mode 100644 index 0000000000000000000000000000000000000000..10212638276b15670ad79fe9a98e472517e6495f --- /dev/null +++ b/checkpoint/backbone/adapter_config.json @@ -0,0 +1,48 @@ +{ + "alora_invocation_tokens": null, + "alpha_pattern": {}, + "arrow_config": null, + "auto_mapping": null, + "base_model_name_or_path": "Qwen/Qwen3-8B", + "bias": "none", + "corda_config": null, + "ensure_weight_tying": false, + "eva_config": null, + "exclude_modules": null, + "fan_in_fan_out": false, + "inference_mode": true, + "init_lora_weights": true, + "kasa_config": null, + "layer_replication": null, + "layers_pattern": null, + "layers_to_transform": null, + "loftq_config": {}, + "lora_alpha": 16, + "lora_bias": false, + "lora_dropout": 0.0, + "lora_ga_config": null, + "megatron_config": null, + "megatron_core": "megatron.core", + "modules_to_save": null, + "monteclora_config": null, + "peft_type": "LORA", + "peft_version": "0.21.0", + "qalora_group_size": 16, + "r": 8, + "rank_pattern": {}, + "revision": null, + "target_modules": [ + "o_proj", + "q_proj", + "v_proj", + "k_proj" + ], + "target_parameters": null, + "task_type": "FEATURE_EXTRACTION", + "trainable_token_indices": null, + "use_bdlora": null, + "use_dora": false, + "use_qalora": false, + "use_rslora": false, + "velora_config": null +} \ No newline at end of file diff --git a/checkpoint/backbone/adapter_model.safetensors b/checkpoint/backbone/adapter_model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..daed7115bd07216c850efb0e7773e7bd986dd743 --- /dev/null +++ b/checkpoint/backbone/adapter_model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3a32ae4c96f6c47365116a7fae31c59b14e102baa825f496bc0291a229f19b1c +size 30707464 diff --git a/checkpoint/checkpoint_manifest.json b/checkpoint/checkpoint_manifest.json new file mode 100644 index 0000000000000000000000000000000000000000..b1fd91d8b887ae779b21d21c3dc291e4870d6810 --- /dev/null +++ b/checkpoint/checkpoint_manifest.json @@ -0,0 +1,14 @@ +{ + "max_length": 8192, + "data_sha256": "11d978325d91cd59a19e2a7a02f78b1afc923b96c40344eeaff413373498e60e", + "validation_data_sha256": "ba8c26010f7cfe2b424821a977afff4e22c44b2b514b827ef80c1f4ed891eca3", + "config_sha256": "dfa6c4110345c3126eb9ce2e8f29d5490acc8515d682175cc2b368b404652268", + "implementation_sha256": "a6ed917fe7e073d31c854483c5f3d4e40d6c7b2f4571fb77bce43bd98aefdf5a", + "backbone": { + "path": "/lus/lfs1aip2/projects/u6uv/shanliantian.u6uv/fastfill-v4/models/Qwen3-8B", + "revision": null, + "config_sha256": "f7c4eadfbbf522470667b797a3c89be2524832d2d599797248dc304fff447c30" + }, + "tokenizer_sha256": "26ef71c7d8e22c3ba858cc760b86da45c755cc6842fb5c2dd14ad4db73e99852", + "step": 6552 +} diff --git a/checkpoint/geometry_model.pt b/checkpoint/geometry_model.pt new file mode 100644 index 0000000000000000000000000000000000000000..0ce343bc4f3a82660391298ccb8667c48c4655ba --- /dev/null +++ b/checkpoint/geometry_model.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f75fe0b391524f5b93ce772eaec3c697fc411f00de429f0d0cd4fc2733ec9651 +size 4773845 diff --git a/checkpoint/model_config.json b/checkpoint/model_config.json new file mode 100644 index 0000000000000000000000000000000000000000..8f797ceb76ec8e66f18d4c5e5f565e2f4d860ac1 --- /dev/null +++ b/checkpoint/model_config.json @@ -0,0 +1,35 @@ +{ + "backbone": "Qwen/Qwen3-8B", + "decoder_dim": 128, + "decoder_heads": 4, + "decoder_layers": 2, + "decoder_ffn_multiplier": 4, + "dropout": 0.0, + "yaw_bins": 12, + "max_objects": 256, + "size_reference": [ + 1.0, + 1.0, + 1.0 + ], + "size_log_limit": 10.0, + "residual_mode": "tanh", + "position_head": "grid_residual", + "position_grid": 16, + "train_backbone": false, + "lora_rank": 8, + "lora_alpha": 16, + "lora_dropout": 0.0, + "lora_target_modules": [ + "q_proj", + "k_proj", + "v_proj", + "o_proj" + ], + "backbone_dtype": "bfloat16", + "local_files_only": false, + "tiny_hidden_size": 64, + "tiny_layers": 1, + "pose_conditioning": false, + "support_head": false +} \ No newline at end of file diff --git a/checkpoint/training_manifest.json b/checkpoint/training_manifest.json new file mode 100644 index 0000000000000000000000000000000000000000..cbcaf4442eb8fa3dcddd8a49128188d2fd599ab2 --- /dev/null +++ b/checkpoint/training_manifest.json @@ -0,0 +1,303 @@ +{ + "data_path": "/lus/lfs1aip2/projects/u6uv/shanliantian.u6uv/fastfill-v4/data/v4fixed/train.jsonl", + "data_sha256": "11d978325d91cd59a19e2a7a02f78b1afc923b96c40344eeaff413373498e60e", + "code_commit": "26939bc46bc57c5a97a154a8b7f7511a68fcba4f", + "code_dirty": true, + "implementation_files_sha256": { + "fastfill/v2/__init__.py": "fa67d3676eafd0076d102f2b16c4f49f40cedbe56c68fb414bc469b2e79c6050", + "fastfill/v2/adapters.py": "d204a1b483f03a1199a8183da565843bf39b274de4eefb7448afa78b594bb4ae", + "fastfill/v2/anchor_probe.py": "ccd7a2e869eede2d19fdb082ba7310c108cd0313a29b2b775a36b96fd0285c46", + "fastfill/v2/audit.py": "6d6cababd3fbec6d1ede58981561aa27b968c9d115ad07738bf5b269059e520b", + "fastfill/v2/autorun.py": "e6826078fee1c774f9e8e1854a22d1cfc2ae309fdd0289ec1cf73b081e289d62", + "fastfill/v2/batch.py": "575a31ac2d82725700f030a0498ea8ab7efc15fc1e5c87cbf53153846171d7fa", + "fastfill/v2/bbox_visualization.py": "65fb0898b1f65f22fa6e9ab4b95b6c0b9369efc22349bf5f267dce1cc5026a09", + "fastfill/v2/boxes.py": "ef0c763fd6584f5b44a2ad8d1976a3f3568b1abf57b8fdcfab562e9c0f384284", + "fastfill/v2/cell_costs.py": "11b1b47e8afea53c3e6a6e12a0ed4b1ce8c23b054471c8a835ac66e16d3555a1", + "fastfill/v2/cohort.py": "7882b18ab09d7443fc854c6ed58100f94247a3cffda8201501f5cf74018ad138", + "fastfill/v2/data.py": "042379618ad979bc51b00dc18f3e31ceafe2b90fc06a0d5215b3fa10cce729e3", + "fastfill/v2/direct_layout.py": "c2230ce930310cceb06d78e4dc2ccee0ab364638a39a2245bd34f08ea02cd801", + "fastfill/v2/evaluate.py": "795a4b60fcca896f61a38f4f6d5cc06bebdb033f8ef37493f29b99b81fdf61de", + "fastfill/v2/geometry.py": "156e3ee5fda5f9d6afc95f2cffda75c760884b25617822df794b97bedc0dae5c", + "fastfill/v2/io.py": "79dd349e8cf4e1e7e842d998fc7190ab792267a28b2142811208692c4c3533b1", + "fastfill/v2/legacy_bridge.py": "ac62037fb3899f684a24bf9d685d13ed9d3f4ca73875fff37b1edd0f337ffced", + "fastfill/v2/legacy_build.py": "aa7f0fbf474707b68187423e1b22c28aacc6b86c0f7a6e0e7eec73a8d5d4328f", + "fastfill/v2/legacy_evidence.py": "a81e5700b356e5723e591dd95c4721cc28a2ad35786e24730cf8c9e6c0326a85", + "fastfill/v2/legacy_verify.py": "3988ed6f89cf3a34c9d0a04310c53cd20fca908eee10a2afe286a9aa7832d527", + "fastfill/v2/llm_baseline.py": "bc50fa57e9250fe86442ed3981c0e9eef4d6388a7303245df0324e0cd97be459", + "fastfill/v2/llm_structured.py": "b0c8f8d1707e392923b49d25f2a4463af8c7a96f548066d34ceaa8b588431dc0", + "fastfill/v2/llm_workflow.py": "155aff6ec182c169f098651e68b5a767e23389328c6cbe990ab06907bc59255f", + "fastfill/v2/losses.py": "d36d5a55fcc2bf69d06da7e2d40bb057c36f87ea6d65413fd451dd75cda65785", + "fastfill/v2/masked_layout.py": "42cec23abe568dcf826b6fe74c902f06375812eb398c10179613185f74fba950", + "fastfill/v2/matching.py": "3ae60074cd184cc3adce3526a7c4582687730c13c018f6a270dc51d49b26dceb", + "fastfill/v2/minimal_data.py": "bbaa70f46e948167040c3215c29bb2c14a65247e3a75dd1000ee78c85116b975", + "fastfill/v2/model.py": "9a615425f6eea41b6d268964fc09e532cc2c18fbc11397d2da9a20ebf699544c", + "fastfill/v2/multisource_data.py": "43c9097979ac410c4ce04a9e2b7c63459f4b26a1bf524ac225219af0b3c7452c", + "fastfill/v2/multisource_verify.py": "881cba1ec548e53010772df685ef9bab2608abeaa6cdad85e3239f2861519230", + "fastfill/v2/objective.py": "f13067db9932940272122ea4320a7eeeea44fe42dbd5c30e320f49c725eefc49", + "fastfill/v2/predict.py": "55f2859836b02fe5f21887ab7141b8f4d9aeab0c3405d97080d2a2e93fb0a46b", + "fastfill/v2/qualified_data.py": "97f298299ac9791282d7712d7b8588f449af738adb833f9e526626d1c1ec0640", + "fastfill/v2/regularizers.py": "8314c07895ae1d39f42e9daf2e906fded13025d2dd9e299eb71651cedc14aaa1", + "fastfill/v2/remask.py": "5b3ddadffc53f0e95c1993a805637d2cac3a8b7949aa0b3d96101aa3a63c5818", + "fastfill/v2/review_data.py": "59c86ec6e5679f442c6a16fb1257febe016310c02fa136d8aff91942afac3184", + "fastfill/v2/roomgenbench.py": "e3cdc050fd86c02243e600bfbd5dfb9c3c57af17c028887647d126dcaf47aea8", + "fastfill/v2/runtime.py": "8bd0c2d7e12531fd6805ce0caaa348be94402e950cf91465ffc22f8ddb4f0993", + "fastfill/v2/schema.py": "d02610ebc6ca57df03f8426c592b50b7b7d7c634f7c3656c41cbd8bddf09a881", + "fastfill/v2/serve.py": "66839f204f4f0a5f12fc794a91e0ea151a9e988f8bf39efa9c046eeccb8bb0b3", + "fastfill/v2/size_range.py": "0ad2e26ea419068f0e92349dd0718ee3c733028711cae2822432d0f4d5b48165", + "fastfill/v2/support.py": "ccd5e6eada67c724111d619f254d3a78b3e33dbe881bf1fd9c0ca27882ca03c1", + "fastfill/v2/text_sft.py": "0dd88d1403a421545942f3aa063a17389dcb914837ee58444ddc58d4bb3a63da", + "fastfill/v2/train.py": "b3dd2e37f8d9cfc5b4c984a46642c84e0cd4a2d23375f3004b562bcb8afebbce", + "fastfill/v2/validation.py": "87ed462441f1a74a0da4c6a16cdca45ee5e6dfd7155239f0a348179e26c99f59", + "fastfill/v2/verify_bundle.py": "812c66c8544567157a3b01e75c84491d6b9886ab75d9826c3a010d97cf30f47a" + }, + "implementation_sha256": "a6ed917fe7e073d31c854483c5f3d4e40d6c7b2f4571fb77bce43bd98aefdf5a", + "packages": { + "torch": "2.13.0+cu126", + "transformers": "5.14.1", + "peft": "0.21.0", + "scipy": "1.15.2", + "accelerate": "1.15.0" + }, + "validation_data_path": "/lus/lfs1aip2/projects/u6uv/shanliantian.u6uv/fastfill-v4/data/v4fixed/validation.jsonl", + "validation_data_sha256": "ba8c26010f7cfe2b424821a977afff4e22c44b2b514b827ef80c1f4ed891eca3", + "backbone": { + "path": "/lus/lfs1aip2/projects/u6uv/shanliantian.u6uv/fastfill-v4/models/Qwen3-8B", + "revision": null, + "config_sha256": "f7c4eadfbbf522470667b797a3c89be2524832d2d599797248dc304fff447c30" + }, + "tokenizer_sha256": "26ef71c7d8e22c3ba858cc760b86da45c755cc6842fb5c2dd14ad4db73e99852", + "config": { + "model": { + "backbone": "/lus/lfs1aip2/projects/u6uv/shanliantian.u6uv/fastfill-v4/models/Qwen3-8B", + "decoder_dim": 128, + "decoder_heads": 4, + "decoder_layers": 2, + "decoder_ffn_multiplier": 4, + "dropout": 0.0, + "yaw_bins": 12, + "max_objects": 256, + "size_reference": [ + 1.0, + 1.0, + 1.0 + ], + "size_log_limit": 10.0, + "residual_mode": "tanh", + "position_head": "grid_residual", + "position_grid": 16, + "train_backbone": false, + "lora_rank": 8, + "lora_alpha": 16, + "lora_dropout": 0.0, + "lora_target_modules": [ + "q_proj", + "k_proj", + "v_proj", + "o_proj" + ], + "backbone_dtype": "bfloat16", + "local_files_only": true, + "tiny_hidden_size": 64, + "tiny_layers": 1 + }, + "loss": { + "position": 1.0, + "size": 0.6, + "yaw_cls": 0.08, + "yaw_reg": 2.0, + "box": 0.0, + "box_operator": "bev_oriented_giou_convex_hull", + "hungarian": true, + "alpha_position": 1.0, + "alpha_size": 1.0, + "collision": 0.0, + "boundary": 0.0, + "position_type": "l1", + "size_type": "l1", + "smooth_l1_beta": 1.0, + "position_cell": 0.5, + "position_residual": 0.4 + }, + "training": { + "steps": 6552, + "batch_size": 1, + "learning_rate": 0.0001, + "gradient_accumulation_steps": 24, + "max_length": 8192, + "seed": 42, + "clip_grad_norm": 1.0, + "mixed_precision": "bf16", + "cpu": false, + "cpu_threads": 1, + "checkpoint_every": 500, + "validate_every": 500, + "resume": null, + "export_model_every_checkpoint": true + }, + "optimizer": { + "warmup_steps": 197, + "schedule": "cosine", + "decoder_lr": 0.0001 + }, + "augmentation": { + "rotate90": true, + "mirror": false, + "shuffle_objects": true, + "drop_constraints_p": 0.3, + "drop_support_p": 0.2, + "category_only_description_p": 0.5, + "minimal_form_p": 0.5 + }, + "validation": { + "exclude_flags": [ + "oob_objects", + "fixed_collision", + "overlapping_furniture" + ] + } + }, + "config_sha256": "dfa6c4110345c3126eb9ce2e8f29d5490acc8515d682175cc2b368b404652268", + "resolved_config": { + "model": { + "backbone": "/lus/lfs1aip2/projects/u6uv/shanliantian.u6uv/fastfill-v4/models/Qwen3-8B", + "decoder_dim": 128, + "decoder_heads": 4, + "decoder_layers": 2, + "decoder_ffn_multiplier": 4, + "dropout": 0.0, + "yaw_bins": 12, + "max_objects": 256, + "size_reference": [ + 1.0, + 1.0, + 1.0 + ], + "size_log_limit": 10.0, + "residual_mode": "tanh", + "position_head": "grid_residual", + "position_grid": 16, + "train_backbone": false, + "lora_rank": 8, + "lora_alpha": 16, + "lora_dropout": 0.0, + "lora_target_modules": [ + "q_proj", + "k_proj", + "v_proj", + "o_proj" + ], + "backbone_dtype": "bfloat16", + "local_files_only": true, + "tiny_hidden_size": 64, + "tiny_layers": 1, + "pose_conditioning": false, + "support_head": false + }, + "loss": { + "position": 1.0, + "size": 0.6, + "yaw_cls": 0.08, + "yaw_reg": 2.0, + "box": 0.0, + "box_operator": "bev_oriented_giou_convex_hull", + "hungarian": true, + "alpha_position": 1.0, + "alpha_size": 1.0, + "collision": 0.0, + "boundary": 0.0, + "position_type": "l1", + "size_type": "l1", + "smooth_l1_beta": 1.0, + "position_cell": 0.5, + "position_residual": 0.4, + "expected_geometry": 0.0, + "door_clearance_m": 0.8, + "support": 0.0 + }, + "optimizer": { + "warmup_steps": 197, + "schedule": "cosine", + "decoder_lr": 0.0001 + }, + "augmentation": { + "rotate90": true, + "mirror": false, + "shuffle_objects": true, + "drop_constraints_p": 0.3, + "drop_support_p": 0.2, + "category_only_description_p": 0.5, + "minimal_form_p": 0.5, + "given_pose_p": 0.0, + "given_pose_corrupt_p": 0.0, + "given_pose_noise_m": 0.0, + "given_pose_yaw_noise_deg": 0.0, + "given_pose_size_noise": 0.0 + }, + "validation": { + "exclude_flags": [ + "oob_objects", + "fixed_collision", + "overlapping_furniture" + ] + }, + "training": { + "steps": 6552, + "batch_size": 1, + "learning_rate": 0.0001, + "gradient_accumulation_steps": 24, + "max_length": 8192, + "seed": 42, + "clip_grad_norm": 1.0, + "mixed_precision": "bf16", + "cpu": false, + "cpu_threads": 1, + "checkpoint_every": 500, + "validate_every": 500, + "export_model_every_checkpoint": true + }, + "max_samples": null, + "dry_run": false + }, + "augmentation": { + "rotate90": true, + "mirror": false, + "shuffle_objects": true, + "drop_constraints_p": 0.3, + "drop_support_p": 0.2, + "category_only_description_p": 0.5, + "minimal_form_p": 0.5, + "given_pose_p": 0.0, + "given_pose_corrupt_p": 0.0, + "given_pose_noise_m": 0.0, + "given_pose_yaw_noise_deg": 0.0, + "given_pose_size_noise": 0.0 + }, + "world_size": 4, + "supervised_samples": 125356, + "rejected_samples": 431, + "validation_samples": 7158, + "validation_minimal_samples": 3737, + "resumed_from": null, + "resumed_at_step": 0, + "resumed_with_changes": {}, + "resume_unverified": [], + "selection_metric": { + "name": "validation.minimal.geometry_objective", + "lower_is_better": true, + "definition": "weighted geometry objective on the minimal (three-field) projection of the validation rows; collapse (BEV overlap rate + central-quarter fraction) is reported beside it; ground_truth anchors collapse_matched (the same objects), never minimised on its own (a degenerate wall-hugging layout scores 0)", + "ground_truth": { + "objects": 50320, + "pairs": 1655488, + "bev_overlap_pairs": 4719, + "same_category_pairs": 121517, + "stacked_same_category_pairs": 2721, + "nearest_wall_distance_sum_m": 27679.385389234172, + "central_quarter_objects": 6549, + "out_of_room_objects": 0, + "footprint_outside_objects": 116, + "floor_objects": 21839, + "door_blocking_objects": 0, + "score": 0.13299757782457322 + } + } +} diff --git a/code/fastfill/README.md b/code/fastfill/README.md new file mode 100644 index 0000000000000000000000000000000000000000..2396bfba20fe7eb7114521b10921de24e8ef5382 --- /dev/null +++ b/code/fastfill/README.md @@ -0,0 +1,342 @@ +# FastFill v1:单次调用的对象条件布局模型 + +仓库入口与版本选择以 [根 README](../README.md) 为准;当前已核实的限制见 [known issues](../docs/known-issues.md)。本页保留 v3/v3.1 的实验说明,数据版本与代码版本须分别记录。 + +v3.2 修复与独立数据版本的说明见 [RELEASE_v3.2.md](RELEASE_v3.2.md)。保留 v3/v3.1 作为原有实验数据;运行中的训练不要切换数据目录。版本号描述数据版本,模型文本协议仍为 FastFill v1。 + +一句话:给定房型、真实房间几何(含已固定在房间里的门、窗、柱子、楼梯等),以及**全部**待摆对象(每件的局部实际尺寸,可选描述与空间约束),模型**一次生成**所有对象的数值位置、yaw 和支撑关系。 + +$$F_\theta(R, G, O, C_{\text{layout}}) \rightarrow L$$ + +2026-09-24 晚改版(数据 v2):不再设物体数上限,落地家具 + 放在家具顶面上的小物件全部保留(含 SAGE),倾斜、悬空的家具修正后保留;门、窗、柱子、楼梯等结构件和没有名字的落地箱子作为 `fixed`(固定件)告诉模型、不摆;挂墙、吊顶、柜子里的东西不摆也不再拒收房间。40960 只是模型的序列长度上限,训练参数由实测数据决定(§4)。 + +整体框架沿用 OptiScene:对象由上游给定,模型只负责摆放,单轮 Qwen3-8B + LoRA SFT。 +在此基础上修正了 OptiScene 已核实的几处缺陷: +- 房间只给面积 → 改为给地面多边形; +- h/w/d 轴序混乱 → 改为局部尺寸,并约定 +X 为正面; +- 同描述物体按列表位置对应 → 每件物体一个 id; +- 训练与推理用的 prompt 不一致 → 改为同一个 prompt。 + +范围和协议以用户冻结的 FastFill v1 定稿为准,接口见附录 B,实现在 `interface.py`。 + +## 1. 流水线与文件 + +``` +HF 导出(22 源) ──download──▶ 本地汇总 JSON ──adapters/*──▶ IR(统一约定) ──build──▶ train/dev/test ──train──▶ LoRA + │ │ + *_rooms.jsonl(评测用 IR) merge_lora ─▶ merged + └──────── evaluate ◀───────┘ +WorldEdge 请求(附录 B) ──interface.request_to_room──▶ 模型一次生成 ──interface.placements_from_text──▶ placements(yaw_rad) +``` + +| 文件 | 作用 | +|---|---| +| `download.py` / `download_full.sh` | 下载导出数据。前者默认只下汇总文件(33 GB),后者用 `hf download` 逐个数据集下全量 | +| `adapters/*.py` | 每个数据源一份,把各自的约定转成统一 IR。每份都附有证据:front test、kits 源码、20 间房逐字段核对 | +| `scene.py` | IR 约定、SYSTEM_PROMPT、序列化、canonical 排序和 id、`parse`(永不抛异常) | +| `anchors.py` | 剔除结构件(`fixed_geometry` 挑出要告诉模型的固定件),推断落地、放在他物上、挂墙、吊顶四类锚点,保证支撑关系闭合 | +| `build.py` | 过滤和拒收规则、固定件、取整对齐、跨源去重、防泄漏划分、增强、约束抽取 | +| `split.py` | 用 house/scan、别名、家具指纹、布局指纹做并查集,按连通分量整体划分 | +| `validate.py` | 越界、支撑、分桶碰撞、OOR 检查,以及约束检查 `holds` | +| `train.py` | LoRA SFT:只监督答案,批量分词,按长度分组,支持 `--resume` 和 `--dry_run` | +| `evaluate.py` | 生成并打分,所有请求都计入统计,分布内和 held-out 分开报告,每项都附 GT 对照 | +| `interface.py` | 附录 B 请求与模型文本之间的双向转换(输出 `position_m` / `yaw_rad` / `support_parent`) | +| `serve.py` | OpenAI 兼容服务端点:给 EmbodiedGen 的 `FastFillBackend` 调用,内部走 `interface.py` | +| `tools/qa_report.py`、`tools/ablation_check.py` | 重建后的 QA(泄漏、闭合、约束、参考布局合法率、固定件统计)和两版只差约束的核对 | +| `merge_lora.py` | 合并 LoRA,保留训练保存的 tokenizer、原始及续训 manifest,并写出 transformers 4.x/5.x 兼容字段 | + +## 2. 统一标准(所有数据源进训练前都转成这一套) + +**坐标与位姿** +- 单位是米,右手系,Z 轴向上。 +- 使用房间局部坐标,边界 AABB 的左下角为 (0,0),地面 z=0。 +- 位置取 bbox **底面中心**。 +- 尺寸是对象**局部坐标**下的 extent `[sx,sy,sz]`,不是世界坐标下的 AABB。 +- 局部 +X 是**语义正面**:训练源须通过正面约定核查;单个对象缺少可靠正面时,按 `no_front` 规则作为固定件或过滤,不要求模型预测它的 yaw。 +- yaw 是绕 +Z 从房间 +X 转到对象正面的角度。 + - 接口层和 IR 用弧度(`yaw_rad`)。 + - 模型文本里写整数度,因为 `90` 是精确值且只占 2 个 token;`1.570796` 不精确,还要多 3–6 个 token。 + - 取整误差 ≤0.5°,对应 1 m 处约 0.9 cm。 + +**房间与对象范围** +- 边界:`polygon` 表示墙体,`hull` 表示扫描地面的凸包。只由物体范围拼出的代理边界不进训练。顶点数不设上限。 +- 待摆对象:落地物体,加上放在其他物体顶面上的物体(带 `on`),所有源一视同仁(SAGE 的桌面小物件也算)。**物体数不设上限**,只要求每间房至少 1 件可摆的物体(只有 1 件的保留并标 `single_object`);唯一的长度限制是模型的 40960 token(超长样本在训练时整条丢弃、不截断,`train.py --dry_run` 会把它们逐条列出来)。 +- 不摆、也不拒收房间的物体:挂墙(画、镜子、挂柜……)、吊顶(吊灯类不管源数据标在多高一律按吊顶)、放在柜子里或不在支撑物顶面上的东西、床上的被子毯子。 +- **固定件 `fixed`**:已经在房间里、模型不摆但必须避开的箱子,连同位姿一起告诉模型(`fixed` 列表:id、size、pos、yaw)。五类(`stats.json` 里按 `fixed:<类>` 计数): + - `structure`(`anchors.fixed_geometry`):门、窗、柱子、楼梯、电梯井、壁炉、暖气片、隔断、栏杆、低梁等结构件,底部低于 1.5 m 的。贴面和无占地的构件(地面、天花、踢脚线、线板、墙板、插座、地漏……)、窗帘百叶纱帘、占房间 ≥80% 的板块不算;`wall` 盒子只有在墙线房间里伸进房间(离墙 10 cm 以外 ≥0.05 m²,即隔断)才算,扫描凸包房间里的 `wall` 一律不算(墙本来就在凸包里)。 + - `generic`:没有名字的落地箱子(`unknown`、`otherprop`、`object`……):它们确实占着地方,但没有名字就没法作为待摆对象(id 为 `object_k`)。 + - `wall`:源数据标成挂墙、但箱子底就在地面上的东西(一整排从地面起的壁柜)。 + - `floating`:墙线(设计师)房间里悬空 ≤0.5 m、下面没有东西撑着的可挂墙家具(浴室柜、悬浮电视柜、搁架):那是挂墙件,不摆,但要避开,保留它真实的高度。扫描源(InternScenes、MultiScan、Scan2CAD)的 z 有噪声,这类家具一律贴地摆放。 + - `no_front`:源数据不给正面朝向的可摆物体(InteriorGS 的定制柜):答案要每件的 yaw,所以把它当嵌入式固定件告诉模型,房间保留(标 `no_front_fixed`),不再整间拒收。 + - 隔断板(`partition panel`、divider、screen)按结构件处理,不当贴面。 + - 统一过滤(`build._keep_fixed`,服务端 `interface.py` 同一套):顶面低于 15 cm 的不算(地垫、踢脚、门把手这类);沉到地面以下的截到地面;整个在房间外的丢;占房间 ≥50% 的丢。**结构件(柱子、楼梯、门、窗……)永远保留**,即使参考布局里有家具压在上面(那会作为参考布局自己的固定件碰撞记入评测的 GT 值);**非结构件**(无名箱子、落地的挂墙件、悬空挂墙件、无朝向件)被已摆物体盖住 ≥50% 时才丢(那是同一件家具的第二个标注框)。 + - 固定件和待摆对象共用一个 id 命名空间(固定件先编号),答案里只出现待摆对象。 +- 倾斜的箱子按"用它的 yaw 的直立包围盒"保留(InternScenes 的 adapter 把倾斜框换成其 8 个角点的直立包围盒;扫描源里主要是书、枕头、杯子、篮子等小件)。落地箱子沉入地面 ≤0.3 m 的贴到地面保留,更深的丢弃;下面没有东西撑着、悬空 ≤0.5 m 的家具:不能挂墙的(床、沙发、桌椅、冰箱……)和扫描房间里的一律贴到地面,设计师房间里的可挂墙家具按 `floating` 固定件处理。放在家具顶面上的画框、镜子按"放在上面"保留,挂在墙上的不摆。 +- **硬过滤(只拒收没法构造训练目标的样本,2026-09-25 起)**:源整体不保证正面朝向;有物体缺包围盒(看不见的障碍);数值不是有限数,或**要摆的**物体尺寸 ≤0(不摆的零厚度贴纸、地垫不再拒收整间房);同一房间里 id 重复;没有边界多边形/凸包,或多边形无效;一件待摆物体都没有。 +- **质量标记(房间保留,写在每条样本的 `flags` 里,`stats.json` 按 `flag:<名>` 计数;`--reject_flagged <名...>` 可以把选中的标记改回拒收)**:`oob_objects`(参考布局有物体出墙 >10 cm 的件数)、`hidden_m2` / `hidden_obstacle`(被丢掉、又不是固定件或贴面的落地箱子露出的地面 m²,≥0.3 时标 hidden_obstacle;主要是楼梯间)、`small_area_m2`(<1 m²)、`room_filling_object`(一件非地毯物体占房间 ≥80%)、`overlapping_furniture`(同类家具大面积重叠或大件硬家具穿插)、`single_object`、`z_snapped`(离地或沉入超过 10 cm 被贴地的件数;每件被贴地的物体都在 IR 里保留原始 z `z_src`)、`tilted`(倾斜换成直立包围盒的件数)。这些是"这条样本的某个字段可不可靠"的标签,不是对数据集的评价;训练时按标签加权或排除是采样层的事。 +- 每个 split 里完全相同的房间(边界相同,且每件物体和固定件的类别、尺寸、位姿都一一相同,允许整体转 90°)只保留一份(3D-FRONT/SpatialLM 同一设计重复最多 26 次;`--keep_duplicates` 可关)。 +- 不再在构建阶段截断大源(`--cap` 默认为空)。采样层在 `train.py`:`--exclude_flags` **默认去掉 `oob_objects`、`overlapping_furniture`、`fixed_collision` 三类**(参考答案是服务端会拒绝的布局:物体出墙 >10 cm;同一件家具重复穿插;家具穿过门、柱子、楼梯等非窗固定件),数据里仍保留,传 `--exclude_flags` 不带值就全用;`--source_weight 源=w` 让某个源的每条样本平均用 w 次(0.5 减半、2 加倍,按 uid 稳定)。启动时打印每个源的条数和去掉的条数;`QA.json` 的 `train rows used by default` 给出默认子集的参考答案合法率。 + +**缺失字段与取整** +- 源数据里缺失的字段一律省略,不伪造。房型、层高缺失时直接不写这个键,训练时也会随机丢掉这两个字段。 +- 物体高过房间层高时,丢弃层高字段。 +- 类别只做可追溯的归一化,原始标签保留在 IR 里;3D-FUTURE 的 'Corner/Side Table'、'Three-Seat / Multi-seat Sofa' 和 Infinigen 的 *factory 名字单独映射成 side table / three seat sofa 等。房型归一化时会去掉实例编号、楼层号和公寓号。 +- 所有校验、约束和评测都用**模型实际看到的取整值**:位置和尺寸取到厘米,yaw 取整数度。生成的约束须在这些参考值上成立;参考布局仍可能有已标记的几何问题,合法率由 `QA.json` 实测,不保证全部样本为 100%。 + +## 3. 数据(v3,2026-09-25 构建:`/Volumes/harddisk/fastfill_data/v3` 和 `v3.1`;HF 数据集 `liantian/fastfill-v3`) + +train 144,140 / dev 8,167 / test 8,647(另有 dev_constrained、test_constrained:每间 1–4 条在参考布局上成立的约束)。 +- v3 = 无约束基线 (R,G,O)→L;v3.1 = 同一批房间,其中 42,743 条训练样本带 1–4 条约束 (R,G,O,C)→L。两版 dev/test 逐字节相同(True);`tools/ablation_check.py`:去掉 constraints 后两版训练样本差异 0 条。 +- `QA.json`:泄漏(train 与 dev/test 之间 group、别名、家具指纹、布局指纹、held-out 源)全部为 0;答案恰好摆了输入里的全部物体 144,140/144,140;固定件出现在答案里 0 次;约束在参考布局上成立 18662/18662(dev)、19785/19785(test)。 +- 参考布局本身的合法率(dev):93.2%。默认训练子集(去掉 `oob_objects`、`overlapping_furniture`、`fixed_collision`)124,836 条,参考答案通过越界、支撑及层高检查的有 124,834 条;另 2 条的支撑距离因厘米取整落到 5 cm 容差边界,旧版严格 contains 判为失败。该比例不衡量源包围盒正确性、语义质量或全部碰撞。模型指标一律和同一批房间的 GT 值对比。 +- token(Qwen3 分词器,`train.py --dry_run`):训练集共 1.86 亿 token;单条 P50 896、P90 2,226、P95 3,440、P99 6,142、最长 135,965;≤4096 的 96.6%,≤8192 的 99.69%。每间待摆物体 P50 6、P90 28、P95 46、P99 84、最长 2,080;固定件 P50 2、P90 7、P95 8、P99 12、最长 70。 +- **超过 40960 的训练样本 9 条**:`interiorgs:0044_839926::room_0`(InteriorGS,2080 件,135,965 token);`interiorgs:0045_839925::room_0`(InteriorGS,680 件,43,739 token);`interiorgs:0134_840039::room_0`(InteriorGS,1274 件,80,252 token);`interiorgs:0141_840177::room_0`(InteriorGS,1575 件,99,434 token);`interiorgs:0175_840149::room_0`(InteriorGS,1229 件,67,984 token);`interiorgs:0179_840114::room_0`(InteriorGS,1164 件,75,057 token);`interiorgs:0394_840186::room_0`(InteriorGS,983 件,57,821 token);`interiorgs:0408_840112::room_0`(InteriorGS,904 件,57,764 token);`MansionWorld::mansionworld/public_entertainment_4f_200_fp001#0/F2_event_hall`(MansionWorld,628 件,44,392 token)。`train.py` 加载时整条跳过、不截断。dev 最长 16,584、test 最长 27,156。 +- 固定件共 477,666 个,分布在 135,205 间写出的房间里(占 84%):structure 441,532、generic 22,040、floating 8,800、no_front 5,128、wall 166。 +- v3 修复了 v2 中的 MansionWorld 桌面物件、InteriorGS 门窗、部分倾斜框、3D-FRONT 精确边界和补门等问题。后续审核仍发现 Structured3D/HSSD/MultiScan 的倾斜框转换及 InteriorGS 上游归房遗漏,修复见 v3.2。v2 / v2.1 的 test 集不同,数字不可直接比较。 + +各源房间数(train / dev / test)、单间最多待摆物体、固定件数、带固定件的房间数(都按写出的房间计): + +| 训练源 | train / dev / test | 最多物体 | 固定件 | 带固定件的房间 | +|---|---|---|---|---| +| SpatialLM | 42,068 / 2,239 / 2,461 | 43 | 156,103 | 46,334 | +| MansionWorld | 19,396 / 1,153 / 1,313 | 628 | 86,316 | 21,762 | +| IL3D_3dfront | 18,054 / 939 / 872 | 27 | 39,427 | 13,567 | +| Structured3D | 12,700 / 793 / 747 | 181 | 73,356 | 14,238 | +| InternScenes_gen | 12,552 / 828 / 660 | 140 | 59,378 | 14,040 | +| SAGE-10k | 8,977 / 503 / 489 | 189 | 12,991 | 9,969 | +| OptiScene_holodeck | 8,946 / 497 / 503 | 21 | 5 | 5 | +| IL3D_synthetic | 6,277 / 349 / 316 | 20 | 1,096 | 999 | +| InteriorGS | 4,867 / 268 / 311 | 2080 | 23,508 | 5,347 | +| InternScenes_arkit | 4,528 / 280 / 229 | 63 | 9,486 | 4,012 | +| HSSD200 | 1,876 / 86 / 138 | 623 | 3,529 | 1,345 | +| InternScenes_scannet | 1,309 / 80 / 104 | 90 | 4,194 | 1,257 | +| InternScenes_mp3d | 1,258 / 89 / 18 | 112 | 5,170 | 1,247 | +| InternScenes_3rscan | 1,200 / 61 / 70 | 160 | 2,588 | 915 | +| MultiScan | 114 / 2 / 6 | 80 | 344 | 101 | +| Scan2CAD | 18 / 0 / 1 | 32 | 0 | 0 | +| SceneSmith(只评测) | 0 / 0 / 364 | 152 | 26 | 25 | +| SpatialGen(只评测) | 0 / 0 / 45 | 25 | 149 | 42 | + +扫描 188,670 间。硬过滤拒收(间):object_count 19,606、boundary 722、boundary_shape 142、incomplete_objects 113。 +- `object_count`:没有一件可摆的物体(只有墙上、顶上的东西或没有名字的箱子),没有训练答案。 +- `boundary` / `boundary_shape`:没有边界或边界多边形无效。`incomplete_objects`:有物体没有包围盒(HSSD 的可动柜门、冰箱等),会成为看不见的障碍。`bad_numbers`:数值非法,或要摆的物体尺寸 ≤0。 +- 另外,每个 split 里完全相同的房间只留一份(duplicate_layout 6,137 间),跨源同一场景只留一个源(dedup_dropped 996 间)。 + +质量标记(写出的房间里,按标记计):tilted 14,073、single_object 13,344、fixed_collision 11,918、oob_objects 10,959、z_snapped 5,059、no_front_fixed 3,480、hidden_m2 2,443、room_filling_object 2,193、overlapping_furniture 1,989、hidden_obstacle 1,257、small_area_m2 59。 + +**数据集取舍:Excel 的 22 个 → 下载保留 16 个(11 个产生训练房间、2 个只做评测、3 个是辅助数据)→ 删除 6 个** + +每个保留的数据集都有实际用途(下面"用在哪"一栏可在 `adapters/*.py` 里对应核实)。房间数见上面 §3 的表。 + +*A. 产生训练房间(11 个下载 → 16 个训练源)* + +| Excel 数据集 | 训练源 | 为什么用 | +|---|---|---| +| SpatialLM | SpatialLM | 设计师户型,官方墙线多边形,房型 100% 有值 | +| IL3D | IL3D_3dfront、IL3D_synthetic | 3D-FRONT 设计房间的规范版本(类别、尺寸完整);程序合成房间 | +| OptiScene | OptiScene_holodeck | 参考项目的数据,只用 Holodeck 生成部分(其中 3D-FRONT 部分与 IL3D 重复) | +| InternScenes | InternScenes_gen;arkit / scannet / mp3d / 3rscan | Infinigen 生成房间(每间都有门窗固定件);四个真实扫描子集,OBB 带语义正面 | +| MansionWorld | MansionWorld | 程序生成楼宇,多边形边界,房型 100% | +| SAGE-10k | SAGE-10k | Agent 生成的单房间,落地家具和桌面小物件都用(v2 起) | +| Structured3D | Structured3D | 官方房间多边形和层高 | +| InteriorGS | InteriorGS | 官方房间 profile 多边形 | +| HSSD-200 | HSSD200 | 设计场景的官方区域多边形;它的 `asset_catalog.csv` 还是 IL3D_synthetic 计算物体网格边界必需的 | +| SceneCAD & Scan2CAD | Scan2CAD | ScanNet 扫描 + CAD 对齐得到的朝向 | +| MultiScan | MultiScan | 真实扫描,标注了正面向量;多数房间没有可用边界(boundary)被拒 | + +*B. 只做评测(held-out,从不进训练)* + +| Excel 数据集 | test | 为什么用 | +|---|---|---| +| SceneSmith | 364 | 训练里没有的生成器,测泛化 | +| SpatialGen | 45 | 同上 | + +*C. 辅助数据:不产生房间,但删掉会让别的数据变错* + +| Excel 数据集 | 被谁用 | 删掉的后果 | +|---|---|---| +| 3D-FRONT | IL3D_3dfront:把 IL3D 按房型合并的房间拆回真实源房间,并取真实地面边界和层高 | 约 1,300 个 IL3D 房间会混着 2–10 个房间的物体,边界也不对 | +| 3RScan | InternScenes_3rscan:reference / rescan 对应表 | 同一空间的多次重扫描可能分到 train 和 test 两边(泄漏) | +| ARKitScenes | InternScenes_arkit:video → visit 对应表 | 同一次 visit 的多段扫描可能分到 train 和 test 两边(泄漏) | + +这三个本身不直接进训练:3D-FRONT 原始导出类别和尺寸大面积缺失,已由 IL3D 版本覆盖;3RScan、ARKitScenes 原始导出没有房间边界,房间来自 InternScenes 版本。 + +*D. 删除(6 个)* + +| Excel 数据集 | 删除原因 | +|---|---| +| HM3DSem | 没有 yaw,尺寸是世界坐标 AABB;region 不是房间 | +| MetaScenes | 没有 yaw(世界坐标 AABB);场景是 ScanNet,已由 InternScenes_scannet 覆盖 | +| ScanNet | 没有 yaw;1,513 个扫描已由 InternScenes_scannet(带正面)覆盖 | +| ScanNet++ | 包围盒没有正面信息(2026-09-24 用数据复核,3 种方法均在对照数据集上先验证有效):X 轴就是较短的水平边(97–100% 的盒子,对照数据集约 50%),即按边长排序而不是按正面;贴墙的一侧在 ±X/±Y 四个方向各约 25%(对照数据集 88–98% 是背面 −X 贴墙);椅子相对桌子的朝向四个方向均匀(n=2,908)。另外 221 个 apartment 未拆成房间,边界是矩形代理 | +| ASE | 约 60% 的实例没有框 | +| M3DLayout | 3D-FRONT、Matterport 部分是重复;Infinigen 部分的边界就是家具范围(3000/3000),不是真实房间 | + +- 删除依据来自各数据集的 HF README 和本地实测(front test、边界检查)。ASE、ScanNet++、M3DLayout_infinigen 转好的 IR 仍保留在 `/Volumes/harddisk/fastfill_ir/`,以后可以直接加进 build;`fastfill.adapters` 遇到已删除的源会跳过并提示。 +- 保留的 16 个数据集已逐文件对照 HuggingFace 核对完整(含 MultiScan 的网格和贴图,共 83 GB)。重新下载用 `fastfill/download_full.sh <目录> <名字...>`(需要 zsh;名字写错或 30 次重试都失败会报错退出)。 + +**许可证(按各数据集本地 README / LICENSE 核对,2026-09-24)** + +| 训练源 | 许可 / 条款 | +|---|---| +| SpatialLM | CC BY-NC 4.0 | +| IL3D_3dfront | IL3D 卡片写 Apache-2.0,但内容是 3D-FRONT:受 3D-FRONT 使用协议约束,只能科研、不能商用、不能用结果提供对外服务、不能分发数据(§3.2、§3.3.1–3.3.5) | +| IL3D_synthetic | 用 HSSD 资产布置,按 HSSD:CC BY-NC 4.0 | +| HSSD200 | CC BY-NC 4.0 | +| InternScenes(gen / arkit / scannet / mp3d / 3rscan) | CC BY-NC-SA 4.0,另加各上游扫描数据集的条款 | +| MultiScan | CC BY-NC 4.0(需申请) | +| Structured3D | 仅非商业科研,不得再分发 | +| InteriorGS | 需申请,按 interiorgs-terms-of-use | +| MansionWorld | CC BY 4.0,但 HF 下载时要勾选"仅非商业科研" | +| SAGE-10k、OptiScene_holodeck | Apache-2.0(约占训练集 17%) | +| Scan2CAD | 本地 README 未写许可;ScanNet 条款为非商业 | +| SpatialGen(评测) | CC BY-NC 4.0 | +| SceneSmith(评测) | Apache-2.0 | + +结论:所有源都允许非商业科研,混在一起训练做内部研究可以;模型继承全部限制,**只能内部非商业使用**。公开权重有冲突(InternScenes 的 ShareAlike 与 3D-FRONT、Structured3D 的禁止分发条款),真要发布需要去掉 InternScenes 或 IL3D_3dfront 重训,并先做法律核查。 + +**防泄漏** +- 以下四种关系任意一种相连的房间,整体划入同一个 split:同一 house 或 scan、同一次 visit、相同家具指纹、相同布局指纹(旋转 90° 视为同一布局,换资产也算)。InternScenes_gen 按房子编号分组(同一个编号出现在不同房型文件夹里就是同一栋房子的不同房间)。 +- 在写出的文件上复核过:train 与 dev/test 之间 group、布局指纹、家具指纹的重合数都是 0;held-out 源在 train 中为 0 行。 + +## 4. 在服务器上训练 + +**环境(CUDA 12.8)**:用 node03 上已验证的组合(torch 2.10 cu128、vLLM 0.19.0、transformers 4.57.6),版本固定在 `requirements-cu128.txt`(在 HF 仓库的 `code/` 下)。训练、合并、推理必须在同一个环境里完成。 + +**历史 v3/v3.1 实验快照**在 HF 数据集 `liantian/fastfill-v3`:`code/`、`v3/`、`v3.1/`;v3.2 使用独立的 `code_v3.2/`、`v3.2/`。复现已有实验使用相应冻结快照。当前 GitHub 安装与入口命令见 [根 README](../README.md)。下面的服务器说明保留原 v3 实验路径。 + +```bash +conda create -n fastfill python=3.11 -y && conda activate fastfill # 已有 node03 的 fastfill 环境就跳过这两行 +pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0 --index-url https://download.pytorch.org/whl/cu128 +pip install -U "huggingface_hub>=0.36" && hf auth login # 用一个 read token +hf download liantian/fastfill-v3 --repo-type dataset --local-dir ~/fastfill-v3 +cd ~/fastfill-v3/code && pip install -r requirements-cu128.txt +python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available(), torch.cuda.device_count())" +python -m fastfill.scene && python -m fastfill.validate && python -m fastfill.interface # 自检 +hf download Qwen/Qwen3-8B --local-dir ~/models/Qwen3-8B +python -m fastfill.train --model ~/models/Qwen3-8B --data ~/fastfill-v3/v3 --out /tmp/x --dry_run # 用真实分词器逐条统计(见下) +``` + +**先测数据,再定参数**(2026-09-24 决定:让数据决定训练参数,不让训练参数决定删哪些家具) +- `--max_len 40960`(默认)只是模型的序列上限,不是显存设置:`train.py` 按每批里最长的那条补齐,显存由数据决定。超过 40960 的样本整条丢弃、不截断。 +- `--dry_run`:用 Qwen 分词器对 train/dev/test 逐条分词,打印每条 token 数、答案 token 数、每间房物体数和固定件数的 P50/P90/P95/P99/最大值,各长度桶的条数,以及超过 `--max_len` 的样本清单(uid、源、token 数、物体数)。超长的极少数拿出来单独讨论,不自动删。 +- `--mem_test 8192,16384,32768,max`:在 1 张卡上(`CUDA_VISIBLE_DEVICES=0`)按训练的真实配置各跑一步(`--bs` 行、按 collator 的方式右填充:第 0 行是该长度,其余行是它的一半,所以 attention mask 也和训练时一样被实体化;梯度检查点和训练一致)。先用 512 token 做一次预热步,再读取 static(权重 + LoRA 参数/梯度 + AdamW 状态);表中每行给出 peak 和这一步新增的部分(激活 / logits / attention mask);某个长度 OOM 时打印后跳到下一个。`max` = train 与 dev 中最长的一行(Trainer 每 `--save_steps` 会在 dev 上评估一次;表前先打印 train / dev 各自最长行的 token 数),数字 = 该长度的合成序列(全部 token 都监督,是上界)。用你打算训练的 `--bs` 跑它,再定 grad_accum。 + ```bash + CUDA_VISIBLE_DEVICES=0 python -m fastfill.train --model ~/models/Qwen3-8B --data ~/fastfill-v3/v3 --out /tmp/x --bs 1 --mem_test 8192,16384,32768,max + CUDA_VISIBLE_DEVICES=0 python -m fastfill.train --model ~/models/Qwen3-8B --data ~/fastfill-v3/v3 --out /tmp/x --bs 2 --mem_test 8192,16384,max + ``` +- 评估固定 `per_device_eval_batch_size=1`;`--eval_max_len N` 把 dev 里超过 N token 的行排除出评估集(打印排除数量,此时 eval loss 只覆盖剩下的子集),N 按 mem_test 的表来选(最长样本见 §3)。 + +**训练**:有效 batch = bs × grad_accum × GPU 数,保持在 32 左右。换 GPU 数量时要同步调整 grad_accum;bs 按 `--mem_test` 的结果定(下面的 bs=4 是 v1 数据 ≤2.8k token 时的值)。 + +```bash +# 8 卡 +PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True torchrun --nproc_per_node 8 -m fastfill.train \ + --model ~/models/Qwen3-8B --data ~/fastfill-v3/v3 --out outputs/ff-v3 \ + --bs 1 --grad_accum 4 --lr 1e-4 --epochs 2 --save_steps 500 # bs 按 --mem_test 的结果改:bs 2 -> grad_accum 2,bs 4 -> 1 +# 只用 1–7 号卡:前面加 CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7,并改 --nproc_per_node 7(有效 batch 28 也可以) +# 单卡:CUDA_VISIBLE_DEVICES=0 python -m fastfill.train ... --grad_accum 8 --save_steps 1000 +# (不指定卡时,单进程会把所有卡用成 DataParallel,又慢又占 0 号卡显存) +# 中断后续训:同一条命令末尾加 --resume。在第 2 个 epoch 中途断开时,该 epoch 剩余部分的数据顺序会变 +# (HF 的已知行为,最多约 1/4 样本在这个 epoch 多训或少训一次),可以接受 +``` + +**资源估算**(v1 数据、bs=4 时的解析估算,v2 的样本更长,以 `--mem_test` 实测为准) +- 2 个 epoch 的用时:8×H100 约 1.5–2 h;8×A100 约 3–4 h(token 总量比 v1 多 39%)。 +- 每个进程都会先在 CPU 上加载一份 16 GB 的权重,8 卡时主机内存峰值约 170 GB。 + +**合并与评测** + +```bash +python -m fastfill.merge_lora --base_model_path ~/models/Qwen3-8B --lora_path outputs/ff-v3/final --output_path outputs/ff-v3-merged +python -m fastfill.evaluate --gt-only --rooms ~/fastfill-v3/v3/test_rooms.jsonl --out eval/gt # GT 参考线 +python -m fastfill.evaluate --model outputs/ff-v3-merged --rooms ~/fastfill-v3/v3/test_rooms.jsonl --out eval/ff-v3 +python -m fastfill.evaluate --model outputs/ff-v3-merged --rooms ~/fastfill-v3/v3/test_constrained_rooms.jsonl --out eval/ff-v3-cons +``` + +**看评测结果时注意** +- 优先装 vLLM;走 HF generate 时设 `CUDA_VISIBLE_DEVICES=0`,否则 8B 模型会被切到多张卡上串行跑;`--batch` 默认 1(40,960 上下文下多于一行的 KV cache 放不进一张卡)。 +- prompt 长度 ≥ `--max_len` 的房间不送模型,记为 `prompt_too_long`,按失败请求计(valid False、约束全 False、parsed False),其余房间照常评。 +- `metrics.json` 的 `in_dist` 和 `held_out` 分开看,碰撞指标一律和同一批房间的 `*_gt` 对比,不要和 1.0 比。 +- 合法(valid)= 不出界、支撑正确、不顶破天花板(物体顶 ≤ 层高 + 5 cm;层高未知时不查)。 +- 越界率、碰撞率、OOR 只在完整解析的答案上算;一个完整答案都没有时显示 `null`(未知),不是 0。 +- **模型指标只看原始输出**(`valid_rate` 等)。`*_repaired` 是接口把物体推回墙内之后的结果,属于确定性后处理,不算模型能力;`inside_1mm_rate` 是按后端 1 mm 标准的贴墙合格率,原始和修复后分开报。 +- **v3(无约束版)训练时没有见过约束输入**,所以 test_constrained 上的约束满足率是"无约束监督基线(零样本)",不能当作约束能力的结论。 +- `rooms_with_fixed_collision`(和 `_gt`)是摆放和固定件重叠的房间比例,对照同一批房间的 GT 值看;序列长度用 `--max_len`(默认 40960,vLLM 的 max_model_len),提示词本身就超过它的房间记为 `prompt_too_long` 失败,不会让整轮评测中断。 + +**起服务接 WorldEdge(EmbodiedGen `FastFillBackend`)** + +```bash +python -m fastfill.serve --model outputs/ff-v3.1-merged --port 8001 # vLLM,贪心解码,一次处理一个房间;--max_len 40960 +# EmbodiedGen 那边:WORLDEDGE_FASTFILL_URL=http://:8001/v1 WORLDEDGE_FASTFILL_MODEL=<服务名,默认是 checkpoint 目录名> +``` + +- 后端把附录 B 请求的 JSON 原样作为 user 消息发过来;`serve.py` 用 `interface.py` 转成模型文本,模型输出后再转回 `placements`(`position_m` / `yaw_rad` / `support_parent`),放在回复的 content 里。 +- 以下情况返回 HTTP 422、不返回布局,由调用方(Harness)决定怎么办:请求不合法(数值非法、坐标系不是米/Z 向上/弧度、重复 id、未知 anchor、约束引用自己或成环、约束引用固定件的 id 等);prompt 超过 `--max_len`(`prompt_too_long`);模型输出解析不了(包括漏摆或多摆物体);修复后仍验收不过(出界、悬空、顶破天花板、有物体超出墙面 1 mm、任何一条 hard 约束不成立)。评测里对应的指标是 `accepted_rate`(同样要求不穿过任何固定件)。FastFill 要么把全部物体合法摆好,要么明确失败,**绝不自己丢物体**。放不下的清单也走这条路。 +- 训练数据里物体常常穿墙几厘米(test 参考答案 39.8% 的房间 > 1 mm,OptiScene_holodeck 系统性 4.5 cm),而后端只容忍 1 mm。所以落地物体穿墙或穿进 keepout 不超过 10 cm 时,`interface.py` 会把它推回墙内,放在它上面的东西一起移动。这是**确定性修复,不算模型成功**:回复里同时给 `raw_placements`(模型原始提案)、`placements`(修复后)和 `validation.raw` / `validation.repaired`(各自的验收结果,含每条约束是否满足),日志和实验三者分开记。 +- 约束字段(这就是后端文档里说的"FastFill v1 文档")。**每个谓词只有一份实现:`validate.holds`**。数据里抽约束、模型 prompt 的定义文字、评测、服务端验收都调用它,所以训练、条件、验收三处语义一致;别的系统要判断 FastFill 约束是否满足,读回复里的 `validation.*.constraints`,不要自己重新实现一遍。 + +| type | 字段 | 定义(`validate.holds`) | +|---|---|---| +| `faces` | `subject`, `target` | subject 的正面方向与"subject 中心 → target 中心"的夹角 ≤ 30° | +| `between` | `subject`, `anchors: [a, b]` | a、b 中心连线与 subject 的旋转 footprint 相交 | +| `supported_by` | `subject`, `parent`, `surface: "top"` | subject 的支撑父物体就是 parent(放在它顶面) | +| `against_wall` | `subject` | subject 旋转 footprint 的某一整条边,每一点离边界都 ≤ 0.1 m | +| `near` | `subject`, `target`, `max_gap_m` | 两个**旋转后 footprint 之间的最短距离** ≤ max_gap_m(间隙,不是中心距离);训练见过 0.1 / 0.2 / 0.3 / 0.5 | +| `keepout` | `polygon_xy`, `hard` | `hard=true`(默认):外扩 0.08 m 后从地面多边形里挖掉;被禁放区围出来的 < 0.5 m² 小死角当作不可用地面。`hard=false`:只报告满足情况,不切地面、不新增模型条件 | + +- 别的系统里叫 near 但定义不同的关系(例如"中心距 ≤ 1.5 m"),**不能**直接改个字段名当作 FastFill 的 `near` 发过来:两者是不同的谓词,需要调用方自己决定怎么对应。 + +- 物体上带 `"anchor": "wall"` 或 `"ceiling"` 的(镜子、挂画、吊灯),v1 不摆,放进 `unsupported` 返回;引用它们的约束也进 `unsupported`,其余物体照常摆。 +- 回复带 `model`(服务名,默认 checkpoint 目录名)和 `checkpoint`(绝对路径);EmbodiedGen 按 `WORLDEDGE_FASTFILL_MODEL` 缓存回复,所以两边名字要一致。422 的回复也带模型原文 `raw_text` 和 `unsupported`。没有可摆物体的请求直接回 200、不调用模型。 +- 训练里从没出现过的约束形式(扫描凸包房间里的 `against_wall`、主语没有语义正面的 `faces`、涉及放在别的物体上的对象的 `near`/`faces`/`between` 等)照样检查,但在 `validation` 里标 `"trained": false`。 +- 禁放区离墙不到 1 cm 时贴到墙上(避免厘米取整后留下一条自相交的细缝);取整后仍然折叠的切口返回 `keepout_not_representable`。 +- 评测的 `metrics.json` 对合法率、验收率、贴墙 1 mm 合格率、出界率都给出同一批房间的参考布局值(`*_gt`)。 +- 请求里可选的 `room.fixed_geometry`:已经在房间里、不能动的箱子(门、窗、柱子、楼梯,或者留在原地的家具),每项 `{"id","category"?,"size_xyz_m","position_m"(底面中心),"yaw_rad"?}`,按 `fixed` 给模型;和它们重叠的摆放记在 `validation.*.fixed_collisions`;其中**穿过门、柱子、楼梯等非窗固定件的**(`fixed_blocking`)判为验收失败(422)。家具挡窗只报告(窗框伸进房间 5 cm,贴墙的柜子、床头常常压到它);家具之间的包围盒重叠(椅子塞桌下)也只报告。推回墙内的修复会连同放在上面的东西一起检查,不会把杯子推进固定柜体。按训练数据同样归一化:|z| ≤ 0.10 归零;底部 ≥ 1.5 m、占地 ≥ 房间一半、或完全在墙外的固定件不给模型,在 `unsupported` 里以 `fixed_ignored` 报告;推回墙内的修复不会把物体更深地压进固定件(找不到不加深重叠的移动就原地不动,由验收判定)。约束不能引用固定件。EmbodiedGen 目前不发这个字段(门口用 keepout),加不加由后端负责人定。 +- 所有约束都按"必须满足"发给模型;验收时 `hard` 默认 true,失败则 422;`"hard": false` 的约束只报告结果,不导致失败。 +- `against_wall` 按真实墙线验收,不算禁放区的切口(模型会把切口看成墙,所以请求里同时有禁放区和 against_wall 时,模型偶尔会因此失败)。 +- keepout 把房间切成两块或在中间留出孤岛时,请求直接报错(`keepout_not_representable`)。 + +## 5. 模型输入输出(真实样本,ARKit 扫描卧室,1 扇窗 + 2 扇门是固定件,6 件待摆,其中 2 盏灯放在柜子上) + +```json +user: {"boundary_type":"hull","boundary":[[0,0.04],[0.04,0],[5.66,0],[5.7,0.04],[5.7,2.93],[5.66,2.97],[0.04,2.97],[0,2.93]], + "fixed":[{"id":"window_1","size":[0.11,1.16,1.23],"pos":[0.2,1.5,0.89],"yaw":356},{"id":"door_1","size":[0.05,0.86,2.1],"pos":[5.33,2.74,0],"yaw":274}, + {"id":"door_2","size":[0.03,0.79,2.03],"pos":[5.52,1.39,0],"yaw":179}], + "objects":[{"id":"bed_1","size":[2.07,1.63,0.61]},{"id":"cabinet_1","size":[0.5,0.84,0.77]},{"id":"cabinet_2","size":[0.52,0.42,0.54]}, + {"id":"cabinet_3","size":[0.53,0.41,0.52]},{"id":"lamp_1","size":[0.29,0.2,0.33]},{"id":"lamp_2","size":[0.21,0.18,0.34]}]} +assistant: {"placements":[{"id":"bed_1","pos":[2.31,1.13,0],"yaw":91},{"id":"cabinet_1","pos":[4.2,2.6,0],"yaw":269},{"id":"cabinet_2","pos":[3.38,0.36,0],"yaw":91}, + {"id":"cabinet_3","pos":[1.26,0.4,0],"yaw":91},{"id":"lamp_1","on":"cabinet_2","pos":[3.38,0.25,0.54],"yaw":90},{"id":"lamp_2","on":"cabinet_3","pos":[1.23,0.24,0.52],"yaw":89}]} +``` + +- `fixed`(可选):已经在房间里、不能动的箱子,带 size / pos / yaw,模型不摆它们、只避开;答案里不出现。 +- 对象按占地面积从大到小排列,id 是 `类别_序号`(固定件先编号,和待摆对象共用命名空间),答案顺序与输入一致。 +- 可选的 `desc` 取资产描述的第一句,已去掉摆放从句,训练时整间房以 50% 的概率丢弃。 +- 可选的 `constraints` 有五种:`["on",a,b]`、`["faces",a,b]`、`["near",a,b,d]`、`["against_wall",a]`、`["between",a,b,c]`。 + +## 6. 重建数据 / 下一步 + +```bash +python -m fastfill.download --out $DATA # 汇总文件 33 GB(全量:fastfill/download_full.sh $DATA) +python -m fastfill.adapters --root $DATA --out $IR # 21 个 adapter → IR(约 20 万个房间) +python -m fastfill.build --ir $IR --out data/fastfill_v3 \ + --sources SpatialLM IL3D_3dfront InteriorGS HSSD200 Structured3D InternScenes_arkit InternScenes_mp3d InternScenes_scannet \ + InternScenes_3rscan Scan2CAD MultiScan OptiScene_holodeck IL3D_synthetic SAGE-10k MansionWorld InternScenes_gen SpatialGen SceneSmith +# v3.1(约束监督):同一条命令加 --constraint_frac 0.3,输出到另一个目录;test 文件不变,可以与 v3 配对比较 +python -m fastfill.tools.qa_report $IR data/fastfill_v3 # -> QA.json(泄漏、闭合、约束、参考合法率、固定件) +python -m fastfill.tools.ablation_check data/fastfill_v3 data/fastfill_v3.1 +``` + +**已知限制** +- **序列长度**:唯一的硬上限是模型的 40960 token;超长样本训练时整条丢弃(`--dry_run` 列出清单)。v3/v3.1 单间待摆物体最多为 InteriorGS 的 2,080 件;超长样本不会被截断后训练。 +- **门窗来源**:SpatialLM、3D-FRONT、MansionWorld、InteriorGS 来自导出里每个房间的 `doors` / `windows`(位置、宽高、窗台高;门厚取自门框轮廓,其余按 0.10 m;被拆成几块的门和飘窗合并成一个,低于 15 cm 的窗台板丢掉;3D-FRONT 只挂在隔壁房间名下的门按几何补给它通向的每个房间,约 62% 的房间有门);Structured3D 来自 `room_context.doors` / `windows`;SAGE-10k 的门来自每个场景的 `layout_*.json`;InternScenes、HSSD、MultiScan、扫描源用自带的门窗标注框(HSSD 元数据生成的门窗框按网格包围盒重新定中心)。Holodeck、IL3D_synthetic 的导出里没有门窗。门窗框放在墙线上,厚墙里(离地面多边形 0.3 m 以内)的也保留;飘窗凸出墙外、不碰墙线的不给模型。 +- **服务端边界**:当前 `clean_boundary` 会迭代厘米取整和简化直到稳定,以保持重复规范化的一致性。历史版本行为应以其冻结代码为准。 +- **MansionWorld 桌面物件**(v3 起读入):导出里有、并和原始 floor JSON 对上的都进来了;AI2-THOR 自带、导出时就没有尺寸的约 2.5 万件(笔记本电脑、盆栽、书……)没有来源尺寸,不编造,不摆。 +- **不在支撑物顶面上的东西**(SAGE 带书架的书桌桌面、柜子里、搁板层上:底部离父物体包围盒顶面超过 5 cm)不摆也不作为固定件:答案格式规定 `on` 的 z 就是父物体的顶面,表示不了它们。SAGE 里约 37% 的带 place_id 的小物件属于这种情况。 +- **支撑关系**:v3/v3.1 统计为 380,269 条源关系、216,025 条推断关系;v3.2 为 380,269 / 216,032。源关系与 bbox 推断都不等于已验证的物理稳定性。 +- **边界**:IL3D_3dfront 在 v3 起用 3D-FRONT 的精确地面多边形(无效时退回地板凸包,`meta` 里记录用了哪种)。 +- **房型标签**:3D-FRONT 自身有房型标错的噪声。 +- **朝向**:InternScenes 存在源正面近乎竖直却仍参与水平 yaw 监督的已确认实例;全库影响比例尚未量化,详见 [known issues](../docs/known-issues.md)。 +- **碰撞检查**:只在 bbox 级别,结果须对照 GT 参考线。 +- **尚未支持**:对称物体 yaw 等价类的训练;把 keepout 也表示成固定件(现在仍是从地面多边形里挖掉)。 +- **许可**:见第 3 节许可证表。训练出的模型只能用于内部非商业研究;不能公开权重,也不能对外提供服务。 diff --git a/code/fastfill/RELEASE_v3.2.md b/code/fastfill/RELEASE_v3.2.md new file mode 100644 index 0000000000000000000000000000000000000000..a56c2dbf3483205f521080eda17148ff93ec110d --- /dev/null +++ b/code/fastfill/RELEASE_v3.2.md @@ -0,0 +1,54 @@ +# FastFill v3.2 release + +Verification: code repairs, full dataset build, independent scan and exact project QA replay passed. The local publication receipt records GitHub commit `5ab744d3de42f0ffc7eadd692cac90dd18d8e392` and Hugging Face commit `d264159e7095e4ce817d46c2fa0bd0cf5dc3d28d`. Repository consolidation preserves these release identifiers; see [versions](../docs/versions.md). + +v3.2 is a separately rebuilt, constrained dataset derived from v3.1 (constraint probability 0.3, the same 18 selected sources). v3 and v3.1 remain immutable baselines for existing training jobs. Do not change a dataset in the middle of a run. + +## Changes + +- Re-express tilted Structured3D, HSSD200 and MultiScan boxes as enclosing upright boxes using their rotated geometry. +- Propagate unknown horizontal fronts for 245 upright MultiScan objects with vertically annotated fronts; their pose and size are unchanged. Separately, 120 tilted MultiScan objects receive corrected envelopes. +- Recover InteriorGS objects that the upstream export left unassigned only when their source box center belongs to exactly one valid repaired room polygon; record recovery decisions and unresolved cases. +- Protect builds against empty/missing inputs, malformed room geometry and partial writes. The output directory must be new or empty; completed output is published by an atomic directory rename. +- Align support tolerance boundaries and singular/plural window handling; enforce API anchor requirements, keep soft keepouts soft, and bound HTTP request reads. +- Validate written-file hashes, training constraints and actual IR-to-message reproduction in QA; compare ablation metadata too. +- Preserve per-prompt generation limits during HF evaluation and adapter tokenizer/run provenance during merging. Download failures return a failing exit code. + +The API now accepts only positive Content-Length values up to 1 MiB; invalid lengths return 400 and oversized requests return 413. Anchor requirements are checked on returned placements; violations return 422. Soft keepouts report satisfaction without altering the floor or adding unseen conditioning fields to the model prompt. + +The text protocol and system prompt remain unchanged. Runtime validation changes can be used with existing weights. New geometry and recovered rooms mean v3.2 is not a constraint-only ablation of v3; compare models on the same versioned evaluation inputs. + +## Existing-run interpretation + +The September 27 evidence identifies 21 retained erroneous boxes across the three adapters. Joining that complete list to the saved v3/v3.1 row index gives 15 boxes (including fixed context boxes) in 10 default-selected training rooms, plus one affected Structured3D test room. This contradicts the claim that only one training object is affected or all affected HSSD rows are excluded. + +The old 124,834 / 124,836 figure measures the implemented out-of-bounds, support and ceiling checks. It is not a general data-purity rate and does not test raw box correctness or all collisions. The known issues do not by themselves justify discarding an ongoing run; the actual server configuration was not inspected here. + +## Reproduction + +Use new writable directories for IR and data; preserve the original versions. The release records source IR hashes, code hashes, build arguments, and every output file hash in MANIFEST.json. Source geometry changes can change deduplication and connected split components; a release comparison must enumerate those changes before cross-version benchmark claims. + +```bash +python -m pytest -q -p no:cacheprovider fastfill/tests +python -m fastfill.merge_lora --base_model_path BASE --lora_path RUN/final --output_path MERGED +``` + +Regression suite: 138 tests and 90 subtests passed, including the tiny real CPU merge, mocked HTTP handlers, source-envelope fixtures and existing synthetic self-checks. Combined line coverage of the 13 changed/new production modules: 90.8% (2,075 / 2,285 executable lines). This is not whole-repository coverage. + +## Dataset verification + +The rebuild scanned 188,670 rooms across 18 sources. Four adapters were rerun for 32,125 IR rooms; the other 14 selected IR files match the v3.1 input hashes. Output: 144,150 train / 8,167 dev / 8,647 test. The default truthy-flag filter selects 124,843 training rows before token-length filtering or source weighting. + +Compared with v3.1, 25 existing train rows and one test row changed; 10 train rows were added and none removed. Split membership is unchanged. Eight recovered InteriorGS rooms pass the default flag filter, while one existing HSSD room now fails it; the net default-selected increase is seven. No new dev/test source group or alias overlaps the old training split. Dev inputs are unchanged; compare old and new models on a common version of the corrected test input. + +The independent scanner uses no project imports. It checked all 25 IR/processed JSONLs, 383,016 rows and 2,251,366,019 bytes. All 15 leakage intersections (five keys across three split pairs) are zero. All 98,808 train, 18,662 dev and 19,785 test constraints hold. Written schemas, reference closure, support cycles, finite numbers and manifest hashes passed. All 124,843 default-selected references pass the scanner's out-of-bounds/support/ceiling checks; this does not assert mesh-level correctness or absence of all collisions. Raw IR retains 90 invalid boundaries, 99 nonpositive sizes and 19 clockwise boundaries; the written schemas pass after preparation. + +All 20 original v3/v3.1 files match their pre-build hashes. The release code snapshot matches all 41 implementation hashes in MANIFEST.json. The separate publisher's seven offline tests passed. Its publication receipt records status `published`; the later consolidation audit could not independently access the remote services. + +The cached Qwen3-8B tokenizer (revision b968826d9c46dd6066d109eabc6255188de91218, offline only) checked all 36 added/modified SFT rows: 575–3,470 tokens including the answer end token, with none above the default 40,960-token limit. This is a targeted length check, not a full-corpus token census; no model weights were loaded. + +Project QA reproduced all 160,964 saved SFT rows exactly from IR (system/user/assistant messages and metadata), with zero mandatory failures. This replay verifies persisted rows; it does not independently rerun the global deduplication and split-selection algorithm. The separate scanner and version comparison check the resulting splits and identities. + +## Subsequent audit + +Request-read timeouts and InternScenes near-vertical semantic-front flags remain open issues; see [known issues](../docs/known-issues.md). The recorded geometry checks and release hashes do not establish that all defects are resolved. diff --git a/code/fastfill/__init__.py b/code/fastfill/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/code/fastfill/adapters/__init__.py b/code/fastfill/adapters/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..7f4c85b893d6cf9f11c0ecacc1b071aa33af251e --- /dev/null +++ b/code/fastfill/adapters/__init__.py @@ -0,0 +1,57 @@ +"""Source adapters: exported dataset files -> IR rooms (see fastfill/scene.py). + +Each module in this package defines + SOURCE = "" # output file name .jsonl + def load(root) -> iterable # yields IR room dicts; root = download dir of fastfill.download + + python -m fastfill.adapters --root /Volumes/harddisk/3D_Room_Collections --out /Volumes/harddisk/fastfill_ir [--only IL3D] +""" +import argparse +import importlib +import json +import os +import pkgutil + + +def registry(): + mods = {} + for m in pkgutil.iter_modules(__path__): + if m.name.startswith("_") or m.name == "unified": + continue + mod = importlib.import_module(f"{__name__}.{m.name}") + if hasattr(mod, "SOURCE") and hasattr(mod, "load"): + mods[mod.SOURCE] = mod + return mods + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--root", required=True) + ap.add_argument("--out", required=True) + ap.add_argument("--only", nargs="*", default=None) + a = ap.parse_args() + os.makedirs(a.out, exist_ok=True) + for name, mod in sorted(registry().items()): + if a.only and name not in a.only: + continue + n = 0 + tmp = f"{a.out}/{name}.jsonl.tmp" + try: + with open(tmp, "w") as f: + for room in mod.load(a.root): + f.write(json.dumps(room, ensure_ascii=False) + "\n") + n += 1 + except FileNotFoundError as e: # dataset not downloaded (or deleted): keep any existing IR file + os.remove(tmp) + print(f"{name}: skipped, input not found: {e.filename}", flush=True) + continue + if n == 0: # an empty input file must not wipe a good IR file + os.remove(tmp) + print(f"{name}: skipped, 0 rooms read; existing IR kept", flush=True) + continue + os.replace(tmp, f"{a.out}/{name}.jsonl") + print(f"{name}: {n} rooms", flush=True) + + +if __name__ == "__main__": + main() diff --git a/code/fastfill/adapters/__main__.py b/code/fastfill/adapters/__main__.py new file mode 100644 index 0000000000000000000000000000000000000000..04aa197cf807ac83c0d2d04da77611d1e40b3421 --- /dev/null +++ b/code/fastfill/adapters/__main__.py @@ -0,0 +1,3 @@ +from fastfill.adapters import main + +main() diff --git a/code/fastfill/adapters/ase.py b/code/fastfill/adapters/ase.py new file mode 100644 index 0000000000000000000000000000000000000000..d0632f56a4c2f34a00765b536d06e966dfedf61d --- /dev/null +++ b/code/fastfill/adapters/ase.py @@ -0,0 +1,46 @@ +"""Aria Synthetic Environments (ASE_exported, 8,373 train scenes / 16,375 wall-ring rooms). + +Export (kits/convert_ase.py): Z-up meters, scene-global; furniture_position = ATEK ts_world_object translation += OBB center; furniture_size = object_dimensions full extents along local X/Y/Z; rotation = ZYX Euler of the +ATEK rotation matrix, z = yaw of local +X. Boundary = closed ring of make_wall endpoints (thickness 0); +the 53 open chains were closed by a guessed straight edge -> dropped here. Front = local -Y (beds touch the +wall with +Y). Category: original object_instances_to_classes string, else the ATEK category name of the +same instance (kept in meta). room_type in the export is inferred from furniture -> kept in meta only. +Limitation: GT exists only for instances seen in the selected ATEK frames (~40% of each scene's instances). +""" +import os + +from fastfill.adapters.unified import convert_room, iter_records + +SOURCE = "ASE" + + +def _atek_name(f): + for rec in (f.get("source_fields") or {}).get("obb3_gt", {}).values(): + if rec.get("category_names"): + return rec["category_names"][0] + return None + + +def load(root): + path = os.path.join(root, "BillLin66__3D_Room_Collections/ASE_exported/ase.jsonl") + for scene in iter_records(path): + sid = scene["scene_id"] # ase__<7-digit index> + for room in scene["rooms"]: + furn, n_atek = [], 0 + for f in room["furniture"]: + if f["furniture_category"] is None: + name = _atek_name(f) + if name and name != "other": + f = {**f, "furniture_category": name} + n_atek += 1 + furn.append(f) + open_chain = bool(room["room_boundary"]) and len(room["room_boundary"]) != len(room["source_fields"]["make_wall"]) + r = {**room, "furniture": furn, "room_type": None, + "room_boundary": None if open_chain else room["room_boundary"]} + yield convert_room(r, source=SOURCE, uid=room["room_id"], group=f"ase:{sid[len('ase_'):]}", + subset=scene["source_subset"], boundary_type="polygon", center_z=True, + front_offset_deg=-90, + meta={"room_type_inferred": room["room_type"], "n_category_from_atek": n_atek, + "open_chain_dropped": open_chain, "front_known": True, + "objects_partial": True}) diff --git a/code/fastfill/adapters/hssd200.py b/code/fastfill/adapters/hssd200.py new file mode 100644 index 0000000000000000000000000000000000000000..441f44beb5af58b6a3330b0428e2bfdb5f32ea9a --- /dev/null +++ b/code/fastfill/adapters/hssd200.py @@ -0,0 +1,424 @@ +"""HSSD-200 (XXXpilar/hssd_clean rebuild of hssd-hab): 168 designer scenes, 2,351 annotated regions. + +Source (README / dataset_manifest.json): meters, Y-up, right-handed, scene-global. objects.csv gives the tight +OBB of every rigid instance: obb_center (box center), obb_half_extents (already scaled) along the asset .glb +axes, obb_rotation_wxyz (world <- glb). regions.csv gives the official region polygon in XZ, floor/ceiling height +and the human region label. Converted here to Z-up by (x, y, z) -> (x, -z, y), z measured from the region floor. + +Object frame: the glb axis that maps to world up is the asset's up. Wall-contact statistics (front test on +beds/toilets/cabinets/wardrobes, also per template) show two authoring frames: up=+Y with front=+Z, and the +same frame turned -90 deg about X (up=-Z, front=+Y). Other up axes (33 objects: 24 standing books, 5 stools, 2 +upside-down carpets ...) have no known front: yaw along a horizontal glb axis, front_known False (build shows them as +built-in fixed boxes); objects whose no axis is vertical (94: books, boxes, pillows) are tilted. Their upright +envelope retains the center and the projected front of the nearest recognized authoring frame; an unrecognized +frame keeps a geometric heading with front_known=False. +Articulated instances (3,626 cabinets, wardrobes, fridges, ...) and 14 rigid rows of templates without a box have no box +and no region in the rebuild; they are assigned to regions by their translation and boxed from their model (URDF, else +asset_catalog / .glb bounds; see _rebuild, tagged box_src). 3,290 are the articulated copy of a boxed rigid row at the +same place (_twin: 3,062 of the same template, 93 a sibling template of the same category, 135 of another category) -> +dropped, counted in meta; 336 are kept. Only rows with no model stay incomplete, so build rejects their room: 1 +articulated (render path '.ao_config.json.glb', no URDF) and the 13 rigid rows of templates 3759 / 3760 / 5804 (absent +from the source, outside every region). is_architectural instances (doors, windows, stairs ...) are kept as "structure" +objects (never placed; a floor staircase rejects the room in build); boxless ones are dropped. Category 'unknown*' +(lexicon id 0) or empty is kept as the label 'unknown*' / 'unknown' (was: incomplete, rejecting 125 rooms): the box is +real, build shows it as a generic fixed box or ignores it when raised. +Boxes lying wholly below the region floor belong to a lower storey without a region at that XY (the source assigns by +XY alone) -> dropped, counted in meta. +Openings boxed from metadata sit off their mesh in the rebuild (half a height too high, some shifted along the wall); +their centre is recomputed from the opening .glb bounds (see _center). +""" +import csv +import functools +import itertools +import json +import math +import os +import struct +import xml.etree.ElementTree as ET +from collections import defaultdict + +import numpy as np +from shapely.geometry import MultiPoint, Point, Polygon + +from fastfill.adapters.internscenes import tilted_bbox +from fastfill.adapters.unified import convert_room + +SOURCE = "HSSD200" +T = np.array([[1, 0, 0], [0, 0, -1], [0, 1, 0]]) # Y-up -> Z-up +FRONT = {(1, 1): (2, 1), (2, -1): (1, 1)} # (up axis, sign) -> (front axis, sign) in glb frame + + +def _rot(q): + w, x, y, z = q + return np.array([[1 - 2 * (y * y + z * z), 2 * (x * y - w * z), 2 * (x * z + w * y)], + [2 * (x * y + w * z), 1 - 2 * (x * x + z * z), 2 * (y * z - w * x)], + [2 * (x * z - w * y), 2 * (y * z + w * x), 1 - 2 * (x * x + y * y)]]) + + +@functools.lru_cache(maxsize=None) +def _glb_bounds(path): + """(lo, hi) of a .glb in its asset frame: every mesh primitive's POSITION min/max through the node transforms + (corners of each primitive box, so exact without node rotation, as in all 841 opening .glbs). None if absent.""" + if not os.path.isfile(path): + return None + with open(path, "rb") as fh: + fh.seek(12) # 12-byte header, then the JSON chunk (length, type) + g = json.loads(fh.read(struct.unpack(" 5 cm vertically (3759-0 + +0.15 m, 3760-x -0.10 m, 3756-x -0.32 m) and 3762-0 doors 0.93 m along the wall. Check against the stage walls + (no stage vertex inside the box shrunk 2 cm in-plane, +-0.3 m deep): doors 98.3% / windows 97.5% clear, vs 83% / + 75% for translation and 3% / 7% for obb_center; door bottoms p10..p90 +0.02 m above the region floor (438 of 467 + within 5 cm, the rest in another storey's region). Without mesh_bounds (.glb missing) the translation is used, + counted in the room meta. + Mesh-sourced rows (all furniture, stairs, fences, the 26 mesh-boxed openings) keep obb_center: box already right.""" + if r.get("bbox_source") != "metadata" or r.get("box_src"): # (rebuilt boxes: see _rebuild) + return json.loads(r["obb_center"]) + t = np.array(json.loads(r["translation"])) + if r.get("mesh_bounds") is None: + return t + lo, hi = r["mesh_bounds"] + return t + _rot(json.loads(r["obb_rotation_wxyz"])) @ (np.array(json.loads(r["non_uniform_scale"])) * (lo + hi) / 2) + + +def _pose(o): + """URDF (rpy about fixed x, y, z) -> 4x4; identity if absent.""" + M = np.eye(4) + if o is not None: + r, p, y = (float(v) for v in o.get("rpy", "0 0 0").split()) + M[:3, :3] = _rot([math.cos(y / 2), 0, 0, math.sin(y / 2)]) @ _rot([math.cos(p / 2), 0, math.sin(p / 2), 0]) \ + @ _rot([math.cos(r / 2), math.sin(r / 2), 0, 0]) + M[:3, 3] = [float(v) for v in o.get("xyz", "0 0 0").split()] + return M + + +@functools.lru_cache(maxsize=None) +def _urdf_bounds(path): + """(lo, hi) of an articulated model in its URDF root frame with every joint at its origin (closed: 3,609 of the + 3,626 instances have all joint positions 0, the rest < 2e-5): each link's visual .glb bounds through the joint + chain. The links are Z-up exports and 1,508 of the 1,509 URDFs turn them Y-up in the root joint (rpy -90 deg about + x), so the root frame is the scene's Y-up frame with the model's origin. None if the URDF or a mesh is missing, or + the root joint does not turn link +Z to +Y (kitchenette0002: its 6 instances are yawed Y-up like their rigid copies, + so that model would lie on its back).""" + if not os.path.isfile(path): + return None + x = ET.parse(path).getroot() + up = {j.find("child").get("link"): (j.find("parent").get("link"), _pose(j.find("origin"))) for j in x.iter("joint")} + root = next(link.get("name") for link in x.iter("link") if link.get("name") not in up) + if not all(np.allclose(M[:3, 2], [0, 1, 0], atol=1e-3) for parent, M in up.values() if parent == root): + return None + pose = lambda link: pose(up[link][0]) @ up[link][1] if link in up else np.eye(4) + pts = [] + for link in x.iter("link"): + for v in link.findall("visual"): + if v.find("geometry/mesh") is None: + continue + b = _glb_bounds(os.path.join(os.path.dirname(path), v.find("geometry/mesh").get("filename"))) + if b is None: + return None + M = pose(link.get("name")) @ _pose(v.find("origin")) + pts.append(np.array(list(itertools.product(*zip(*b)))) @ M[:3, :3].T + M[:3, 3]) + pts = np.vstack(pts) + return pts.min(0), pts.max(0) + + +def _rebuild(r, lo, hi, src): + """Box a boxless row in place from its model bounds [lo, hi] in the frame its rotation applies to: centre + translation + R (s * (lo + hi) / 2), half extents |s * (hi - lo)| / 2, box_src = src. + Articulated rows: the URDF root frame (_urdf_bounds). Their translation_origin says 'COM' but the translation is + the root origin (the root link has no mass): on the 1,480 unit-scale rows of up=+Y templates with a same-template + rigid copy within 10 cm, the rebuilt centre minus the copy's obb_center equals the two translations' difference + within 1 cm for 99.0%, the world half extents agree within 1 cm for 99.4%. For templates authored up=-Z the render + .glb (asset_catalog bounds) is not that frame: it is Z-up and centred, the URDF links are re-exported (Cabinet0004: + .glb z +-0.72, URDF base link z 0..1.45), so catalog bounds would lay their 467 yawed articulated rows on their + back; on 118 of them with an overlapping unit-scale rigid copy the URDF box has the copy's extents (p90 diff 0, + catalog p50 0.18 m) and centre (horizontal p50 2 cm; vertically the source places them 0.1-0.3 m off each other). + 26 articulated rows are in the render .glb frame (asset_catalog): 20 carry the rigid copy's up=-Z rotation (root +Y + not vertical) and 6 are kitchenette0002 (see _urdf_bounds); so are the rigid rows (asset_catalog, else the .glb). + Kept rebuilt boxes pass the wall-contact front test like source boxes (IR floor cabinets, wardrobes, dressers, + fridges, nightstands within 15 cm of a wall: back to the wall 136 of 141 vs 2,581 of 2,714); of the 326 in rooms, + 219 stand within 5 cm of the floor and 94 are above 0.5 m (wall units).""" + s = np.array(json.loads(r["non_uniform_scale"])) + c = np.array(json.loads(r["translation"])) + _rot(json.loads(r["obb_rotation_wxyz"])) @ (s * (lo + hi) / 2) + r.update(obb_center=json.dumps(c.tolist()), obb_half_extents=json.dumps((abs(s * (hi - lo)) / 2).tolist()), + box_src=src) + + +def _model(r, base, catalog): + """(lo, hi, src) of a boxless row's model in the frame its rotation applies to (see _rebuild), or None: the URDF + when the row's rotation keeps its root +Y up, else the asset_catalog bounds, else the render .glb.""" + if r["urdf_path"] and abs(_rot(json.loads(r["obb_rotation_wxyz"]))[1, 1]) > 0.999: + b = _urdf_bounds(os.path.join(base, "meshes", r["urdf_path"])) + if b: + return (*b, "urdf") + b = catalog.get(r["template_name"]) + if b: + return (*b, "catalog") + b = _glb_bounds(os.path.join(base, "meshes", r["render_asset"])) + return b and (*b, "glb") + + +def _box(r): + """(XZ footprint, y0, y1) of a boxed row's OBB in the source frame (footprint = hull of the 8 corners).""" + p = np.array(list(itertools.product((-1, 1), repeat=3))) * json.loads(r["obb_half_extents"]) + p = p @ _rot(json.loads(r["obb_rotation_wxyz"])).T + _center(r) + return MultiPoint(p[:, [0, 2]].tolist()).convex_hull, p[:, 1].min(), p[:, 1].max() + + +def _twin(a, b, rel): + """True if boxed row b (from _box) is the piece of furniture that rebuilt box a enters again: they share over 30% of + the smaller one's volume (as build's overlapping_furniture test: two solid pieces cannot) and, by how b's row + relates to a's (rel, see _rel): 'model' (same template): always, the rigid row may be scaled down (8 rows at + 0.23-0.33 of the volume: Cullen Chest, Carla Wardrobe, ...); 'category' (same category): unless b is a small part + inside a (under a third of its volume); 'other' (b not architectural, of a category the source ships articulated + models of): only at similar size (volumes within 3x); None: never. + The articulated copy differs from its rigid row in scale (articulated rows are unit scale), pivot, translation + (same-size kitchen units 0.2-0.5 m apart: Oven 0.37 m) and often template (Raleigh Wide / Tall Dresser), then often + in category too ('Kitchen tall cabinet' wardrobe vs '... with oven' cabinet, Gretna Base Cabinet Double + kitchen_lower_cabinet vs Gretna Drawers cabinet, Melbourne vs Xena 3 Door Wardrobe). Of the 479 rebuilt boxes the + same-category test keeps, 150 share over 30% with a similar-size box of another category (3.1% of the rigid rows of + the articulated categories); 'other' takes 135 (rigid rows 2.5%). The 13 left share with a plant, lamp, TV, toilet, + cooker hood, basket, toy bin or child's bed. In build output the placed rebuilt objects that share over 30% with a + similar-size placed object of another category go from 135 of 360 to 9 of 227 (source objects 173 of 5,911). Also + taken: a desk and two islands whose partner is a chair / stools / a wine cart tucked under them (seat and cart have + articulated models: storage benches, trolleys).""" + if rel is None: + return False + (fa, a0, a1), (fb, b0, b1) = a, b + va, vb = fa.area * (a1 - a0), fb.area * (b1 - b0) + return fa.intersection(fb).area * max(0.0, min(a1, b1) - max(a0, b0)) > 0.3 * min(va, vb) and \ + (rel == "model" or vb >= va / 3 and (rel == "category" or vb <= 3 * va)) + + +def _rel(r, o, kinds): + """How boxed row o relates to boxless row r, for _twin (kinds: the categories of articulated rows). Not the asset + name: collections share it ('Champagne' bed and nightstand, 'Kitchen Furniture Collection' modules).""" + if o["template_name"] == r["template_name"]: + return "model" + if o["category"] == r["category"]: + return "category" + return "other" if o["category"] in kinds and o["is_architectural"] != "True" else None + + +def _furniture(r, floor): + """objects.csv row -> unified-schema furniture dict (Z-up, center z above the region floor).""" + cat = r["category"] or "unknown" # lexicon id 0 ('unknown', 'unknown_wall') or none: no name + f = {"furniture_category": cat, "furniture_instance_id": f"{r['scene_id']}:{r['instance_id']}", + "furniture_position": None, "furniture_rotation": None, "furniture_size": None} + if not r["obb_center"]: + return f # articulated / missing template: no box + C = T @ _rot(json.loads(r["obb_rotation_wxyz"])) + ext = 2 * np.array(json.loads(r["obb_half_extents"])) + c = T @ np.array(_center(r)) + k = int(np.argmax(abs(C[2]))) + f["furniture_position"] = {"x": c[0], "y": c[1], "z": c[2] - floor} + fr = FRONT.get((k, int(np.sign(C[2, k])))) + if fr is None: # unknown authoring frame: the box is known, its front not + fr, f["front_known"] = (min({0, 1, 2} - {k}), 1), False + if abs(C[2, k]) < 0.999: # no vertical axis: tilted + a, s = fr + order = [a, 3 - a - k, k] + R = C[:, order] * np.array([s, 1, 1]) + size, yaw, _, _ = tilted_bbox(R, ext[order]) + return {**f, "furniture_rotation": {"x": None, "y": None, "z": yaw}, + "furniture_size": dict(zip(("width", "length", "height"), size)), + "rot_wxyz_yup": json.loads(r["obb_rotation_wxyz"])} + a, s = fr + o = 3 - a - k + d = s * C[:, a] + f["furniture_rotation"] = {"x": 0, "y": 0, "z": math.degrees(math.atan2(d[1], d[0]))} + f["furniture_size"] = {"width": float(ext[a]), "length": float(ext[o]), "height": float(ext[k])} + return f + + +def load(root): + base = os.path.join(root, "XXXpilar__hssd_clean") + regions = {} + for g in csv.DictReader(open(os.path.join(base, "regions.csv"))): + g["poly"] = [[x, -z] for x, z in json.loads(g["poly_loop_xz"])] + regions[(g["scene_id"], g["region_id"])] = g + by_scene = defaultdict(list) + for key, g in regions.items(): + by_scene[key[0]].append((key, Polygon(g["poly"]), float(g["floor_height"]), float(g["ceiling_height"]))) + + bounds = {k["template_name"]: (np.array(json.loads(k["local_bbox_min"])), np.array(json.loads(k["local_bbox_max"]))) + for k in csv.DictReader(open(os.path.join(base, "asset_catalog.csv"))) if k["local_bbox_min"]} + rows, boxed = list(csv.DictReader(open(os.path.join(base, "objects.csv")))), defaultdict(list) + kinds = {r["category"] for r in rows if r["instance_kind"] == "articulated"} + for r in rows: + if r["bbox_source"] == "metadata" and r["obb_center"]: + r["mesh_bounds"] = _glb_bounds(os.path.join(base, "meshes", r["render_asset"])) + if r["obb_center"]: + boxed[r["scene_id"]].append((_box(r), r)) + furn, n_arch, n_below, n_nob, n_twin = defaultdict(list), *(defaultdict(int) for _ in range(4)) + for r in rows: + boxless, twin = not r["obb_center"], False + if boxless and (m := _model(r, base, bounds)): + _rebuild(r, *m) + a = _box(r) + twin = any(_twin(a, b, _rel(r, o, kinds)) for b, o in boxed[r["scene_id"]]) + if r["region_source"] in ("inside", "nearest"): + keys = [(r["scene_id"], r["region_id"])] + # the source assigns by XY only when one polygon matches: boxes wholly below this region's + # floor stand on a lower storey that has no region there -> not in this room (aabb shifted like _center) + if r["obb_center"] and json.loads(r["aabb_max"])[1] - (json.loads(r["obb_center"])[1] - _center(r)[1]) \ + < float(regions[keys[0]]["floor_height"]) - 0.05: + n_below[keys[0]] += 1 + continue + elif boxless: # no region in the source: locate by translation + x, y, z = json.loads(r["translation"]) + keys = [k for k, p, fl, ce in by_scene[r["scene_id"]] if p.contains(Point(x, -z)) and fl - 0.3 <= y <= ce] + else: + continue # outdoor props outside every region + for key in keys: + if twin: # its rigid copy carries the box already + n_twin[key] += 1 + continue + n_nob[key] += "mesh_bounds" in r and r["mesh_bounds"] is None + f = _furniture(r, float(regions[key]["floor_height"])) + if r["is_architectural"] == "True": + # kept as "structure" (never placed) so build sees floor stairs; without a box it cannot + # be checked and would read as an incomplete object: dropped and counted, as before + if f["furniture_rotation"] is None: + n_arch[key] += 1 + continue + f["structure"] = True + f["box_src"] = r.get("box_src") + furn[key].append(f) + + for key, g in regions.items(): + room = {"room_id": f"{key[0]}:{key[1]}", "room_type": g["room_type"], "room_boundary": g["poly"], + "room_height": float(g["extrusion_height"]), "furniture": furn[key]} + yield convert_room(room, source=SOURCE, uid=f"{SOURCE}:{key[0]}:{key[1]}", group=f"hssd:{key[0]}", + boundary_type="polygon", center_z=True, + extra=lambda f: {**({"rot_wxyz_yup": f["rot_wxyz_yup"]} if "rot_wxyz_yup" in f else {}), + **({"structure": True} if f.get("structure") else {}), + **({"box_src": f["box_src"]} if f.get("box_src") else {}), + **({"front_known": False} if f.get("front_known") is False else {})}, + meta={"region_name": g["region_name"], "floor_height": float(g["floor_height"]), + "n_architectural_dropped": n_arch[key], "n_below_floor_dropped": n_below[key], + "n_articulated_twin_dropped": n_twin[key], + "n_opening_no_mesh_bounds": n_nob[key], "front_known": True}) + + +if __name__ == "__main__": + # glb frame turned -90 deg about X (up=-Z), yawed 90 deg in the scene: front must point to world +Y (Z-up) + q_x = [math.cos(math.pi / 4), math.sin(math.pi / 4), 0, 0] # Rx(+90): glb -Z -> +Y up + R = _rot([math.cos(math.pi / 4), 0, math.sin(math.pi / 4), 0]) @ _rot(q_x) # then yaw 90 about +Y + w = math.sqrt(1 + np.trace(R)) / 2 + q = [w, (R[2, 1] - R[1, 2]) / (4 * w), (R[0, 2] - R[2, 0]) / (4 * w), (R[1, 0] - R[0, 1]) / (4 * w)] + row = {"category": "toilet", "scene_id": "s", "instance_id": "0", "obb_center": "[1, 0.4, -2]", + "obb_half_extents": "[0.2, 0.35, 0.4]", "obb_rotation_wxyz": json.dumps(q)} + f = _furniture(row, 0.0) + # canonical front +Z yawed 90 about +Y -> Habitat +X -> Z-up +X; sizes: front 0.7, side 0.4, up 0.8 + assert abs(f["furniture_rotation"]["z"]) < 1e-6, f + assert np.allclose([f["furniture_size"][k] for k in ("width", "length", "height")], [0.7, 0.4, 0.8]), f + assert np.allclose([f["furniture_position"][k] for k in "xyz"], [1, 2, 0.4]), f + # metadata-boxed opening: obb_center is one half height too high, translation is the true centre + door = {"category": "door", "scene_id": "s", "instance_id": "1", "bbox_source": "metadata", + "obb_center": "[1, 2.2, -2]", "translation": "[1, 1.1, -2]", "obb_half_extents": "[0.5, 1.1, 0.1]", + "obb_rotation_wxyz": "[1, 0, 0, 0]"} + f = _furniture(door, 0.0) + assert np.allclose([f["furniture_position"][k] for k in "xyz"], [1, 2, 1.1]) and f["furniture_size"]["height"] == 2.2, f + # ... unless its mesh bounds are known: mesh x in [-0.5, 2.5] (centre +1.0), y in [-0.6, 1.6] (centre +0.5), + # mirrored in x (s = -1) and yawed 90 deg about +Y: offset R (s * c) = Ry90 (-1, 0.5, 0) = (0, 0.5, 1) Y-up + door.update(mesh_bounds=(np.array([-0.5, -0.6, -0.1]), np.array([2.5, 1.6, 0.1])), non_uniform_scale="[-1, 1, 1]", + obb_half_extents="[1.5, 1.1, 0.1]", obb_rotation_wxyz=json.dumps([math.cos(math.pi / 4), 0, + math.sin(math.pi / 4), 0])) + f = _furniture(door, 0.0) + assert np.allclose([f["furniture_position"][k] for k in "xyz"], [1, 1, 1.6]), f # (1, 1.6, -1) Y-up + # _glb_bounds: minimal .glb, one node translated (+1, 0, 0) and scaled 2 in y over a unit-cube accessor + import shutil + import tempfile + tmp = tempfile.mkdtemp() + js = json.dumps({"scenes": [{"nodes": [0]}], "nodes": [{"mesh": 0, "translation": [1, 0, 0], "scale": [1, 2, 1]}], + "meshes": [{"primitives": [{"attributes": {"POSITION": 0}}]}], + "accessors": [{"min": [-0.5, 0, -0.5], "max": [0.5, 1, 0.5]}]}).encode() + with open(os.path.join(tmp, "body.glb"), "wb") as fh: + fh.write(struct.pack("<4sII", b"glTF", 2, 20 + len(js)) + struct.pack(" (x, z, -y): base link x 0.5..1.5, + # y -0.5..0.5, z -2..0; a child link on a joint 1 up the base's +Z = root +Y: y 0.5..1.5. Union y -0.5..1.5 + urdf = lambda rpy, mesh: f""" + + + + + """ + for name, rpy, mesh in (("ok", -math.pi / 2, "body.glb"), ("flat", 0, "body.glb"), ("miss", -math.pi / 2, "x.glb")): + with open(os.path.join(tmp, name + ".urdf"), "w") as fh: + fh.write(urdf(f"{rpy} 0 0", mesh)) + lo, hi = _urdf_bounds(os.path.join(tmp, "ok.urdf")) + assert np.allclose(lo, [0.5, -0.5, -2]) and np.allclose(hi, [1.5, 1.5, 0]), (lo, hi) + assert _urdf_bounds(os.path.join(tmp, "flat.urdf")) is None and _urdf_bounds(os.path.join(tmp, "miss.urdf")) is None + shutil.rmtree(tmp) + # _rebuild: articulated cabinet, origin at its bottom centre, yawed 90 deg about +Y (front +Z -> +X), box from the + # model bounds; _center keeps the rebuilt centre even for a 'metadata' row; _furniture then as for rigid rows + y90 = json.dumps([math.cos(math.pi / 4), 0, math.sin(math.pi / 4), 0]) + cab = {"category": "", "scene_id": "s", "instance_id": "2", "bbox_source": "metadata", "obb_center": "", + "translation": "[1, 0, -2]", "obb_rotation_wxyz": y90, "non_uniform_scale": "[1, 1, 1]"} + _rebuild(cab, np.array([-0.3, 0, -0.2]), np.array([0.3, 0.9, 0.2]), "urdf") + assert np.allclose(_center(cab), [1, 0.45, -2]) and cab["box_src"] == "urdf", cab + f = _furniture(cab, 0.0) + assert f["furniture_category"] == "unknown" and abs(f["furniture_rotation"]["z"]) < 1e-6, f # '' -> 'unknown' + assert np.allclose([f["furniture_size"][k] for k in ("width", "length", "height")], [0.4, 0.6, 0.9]), f + assert np.allclose([f["furniture_position"][k] for k in "xyz"], [1, 2, 0.45]), f + # unknown authoring frame (glb +X up): box kept, yaw along glb +Y (-> Y-up -X -> Z-up yaw 180), front_known False + book = {**row, "category": "unknown_wall", "obb_rotation_wxyz": json.dumps([math.cos(math.pi / 4), 0, 0, + math.sin(math.pi / 4)])} + f = _furniture(book, 0.0) + assert f["front_known"] is False and f["furniture_category"] == "unknown_wall", f + assert abs(f["furniture_rotation"]["z"] % 360 - 180) < 1e-6, f + assert np.allclose([f["furniture_size"][k] for k in ("width", "length", "height")], [0.7, 0.8, 0.4]), f + # _twin: a same-size unit 0.37 m off (38% shared) is the same piece; a neighbour touching it or a small part + # inside it is not, unless it is the same model (scaled-down rigid row); across categories only at similar size + box = lambda x, h: _box({"obb_center": json.dumps([x, h, 0]), "obb_half_extents": json.dumps([0.3, h, 0.3]), + "obb_rotation_wxyz": "[1, 0, 0, 0]"}) + small = _box({"obb_center": "[0, 0.1, 0]", "obb_half_extents": "[0.1, 0.1, 0.1]", + "obb_rotation_wxyz": "[1, 0, 0, 0]"}) + assert _twin(box(0, 0.45), box(0.37, 0.45), "category") and not _twin(box(0, 0.45), box(0.6, 0.45), "category") + assert not _twin(box(0, 0.45), small, "category") and _twin(small, box(0, 0.45), "category") + assert _twin(box(0, 0.45), small, "model") and _twin(box(0, 0.45), box(0.37, 0.45), "other") + assert not _twin(box(0, 0.45), small, "other") and not _twin(small, box(0, 0.45), "other") + assert not _twin(box(0, 0.45), box(0, 0.45), None) + # _rel: same model > same category > another category with articulated models (not architectural) > none + cab = {"template_name": "t1", "asset_name": "Gretna Base", "category": "kitchen_lower_cabinet"} + rel = lambda **o: _rel(cab, {**cab, "template_name": "t2", "asset_name": "", "is_architectural": "False", **o}, + {"cabinet", "kitchen_lower_cabinet"}) + assert rel(template_name="t1") == "model" and rel(asset_name="Gretna Base") == "category" + assert rel(category="cabinet") == "other" and rel(category="cabinet", asset_name="Gretna Base") == "other" + assert rel(category="plant") is None and rel(category="cabinet", is_architectural="True") is None + print("hssd200.py self-check ok") diff --git a/code/fastfill/adapters/il3d_3dfront.py b/code/fastfill/adapters/il3d_3dfront.py new file mode 100644 index 0000000000000000000000000000000000000000..a2488c7e60b301cd2ee29945210bcaa5a7443262 --- /dev/null +++ b/code/fastfill/adapters/il3d_3dfront.py @@ -0,0 +1,394 @@ +"""IL3D, 3D-FRONT subset (20,274 designer rooms): the canonical copy of 3D-FRONT rooms. + +Verified conventions (scratch checks against IL3D's own USDZ meshes and the 3D-FRONT export): +- position = the 3D-FUTURE asset origin (IL3D copies the raw 3D-FRONT `pos`); every sampled mesh has its origin + at the bottom center (y_min = 0, x/z centered), so it is the IR bottom center. XY are room-local (IL3D moved each + room so its own floor mesh AABB starts at 0). +- size: the layout `bbox` is NOT usable as-is: for int-scale placements it is [x, z, y] of the mesh, otherwise + [x, y, z] unscaled, and the IL3D export reads it as [w, h, l] for all. assets.json meta_data width/length/height + equal the mesh extents along x / y(up) / z for all sampled assets, and the official USD script applies `scale` in + the asset frame, so the placed local extents are meta x |scale|. +- rotation: the export's furniture_rotation.z = yaw of mesh +X (from the full rotateXYZ matrix; tilted -> x/y null). + 3D-FUTURE meshes face +Z (y-up) = IR local -Y, hence front_offset_deg = -90. Only x-mirrors (scale x < 0) occur, + which do not flip the front. +- tilted placements (76, x/y null; 2 of them also z null): the export keeps the full world-from-local Z-up matrix + (source_fields.rotation_matrix_zup; = the raw 3D-FRONT quaternion within 0.014 for all 76, IL3D rounds to whole + degrees), local X / Y / Z = mesh x / -z / y(up) = placed_size's width / length / height, origin = bottom centre. + `untilt` turns them into upright boxes by the internscenes.canon rules (see there): 44 (tilt p10/p50/p90 7/11/31 + deg) become the yaw-aligned bounding box of their 8 corners (flagged tilted, tilt_deg), 32 (22 lying on a side, 10 + tilted <= 3 deg) the exact box with permuted sizes; the 2 z-null ones (a 1 mm beam) no longer make 2 rooms incomplete. +- rooms: IL3D groups a house's objects by room TYPE, so 1,300 IL3D "rooms" merge objects of 2-10 source rooms + (e.g. three bedrooms in one), and its single floor rectangle may belong to yet another room. Every IL3D object is + matched to its 3D-FRONT export placement (same asset id and position under one per-room frame shift), which gives + its true source room; each IL3D room is split into one IR room per source room. +- boundary / height: the exact floor polygon of the source room: the union of its Floor-mesh triangles from the raw + house JSON (data/3D-FRONT/.json, same [x, y, z] -> [x, -z] frame as the export), moved into IL3D's frame by + the shift, boundary_type "polygon". The export's room_boundary is only the convex hull of those vertices (its README: + "not the exact floor plan"). Its area is within 5% of the source room size for 13,496 of the 13,859 rooms that state + one (the hull: 8,085). Gaps and spikes narrower than SPIKE_M (seams between meshes, zero-width slivers of + degenerate triangles, up to 12 cm long) are closed / cut off, pieces and holes < ARTEFACT_M2 dropped; a floor that + is still several pieces (21 rooms), has a real hole (20, stair voids), or whose ring is invalid as is or on the + model's cm grid (`cm_valid`, 4) keeps the export hull, boundary_type "hull" (meta.boundary_from says which). + Height: the export's floor-to-ceiling height (IL3D's supplemented height when the export has none and the room is + IL3D's own floor room). +- anchors: assets.json placement flags are per ASSET, so they are used only when a single flag agrees with z. +- doors / windows: the 3D-FRONT export lists them per source room (room["doors"] / ["windows"]: position = opening + centre, width, height, sill_height (windows), profile = the door's floor footprint [[x, y] x4] incl. the frame + depth, or the window's vertical quad [[x, y, z] x4]); emitted as "structure" boxes in IL3D's frame (export minus + the frame shift): local X along the wall = width (the profile side closest to it gives the wall direction), depth = + door footprint area / width, 0.10 m for windows, bottom = sill_height or centre - height / 2. The export lists a + door on only one of the two rooms it joins (1 of 5,999 door positions in the first 1,500 houses is on two lists), + so a door listed on another room of the house is added when it opens into this one (`on_edge`, tagged listed_on): + 10,768 doors, rooms with a door 7,470 -> 13,465 of 21,832; doors > 0.3 m inside the boundary 5,165 -> 19. + The export splits one door / bay window into several mesh pieces (its README: "the entry count is not the number + of real doors/windows"): pieces lower than MIN_H (sill slabs) are dropped, and a piece whose footprint is >= half + covered by a bigger piece of the same kind is merged into it (dropped), so each opening is shown once. +""" +import collections +import json +import math +import os + +import numpy as np +from shapely.geometry import LinearRing, Polygon +from shapely.ops import unary_union + +from fastfill.adapters.internscenes import tilted_bbox, upright +from fastfill.adapters.unified import _num, convert_room, iter_records +from fastfill.scene import clean_boundary, footprint + +OPENING_T = 0.10 +MIN_H = 0.15 # = build.FLAT_Z: a lower piece is a sill slab, not an opening +ARTEFACT_M2 = 0.01 # floor pieces / holes smaller than this are triangulation slivers (seen: <= 0.0005 m2) +SPIKE_M = 0.02 # narrower gaps / spikes (mesh seams, degenerate triangles) do not survive the model's cm grid +DOOR_TOL = 0.05 # a door footprint ends within 5 cm of the floor edges it opens onto +LEVEL_TOL = 0.3 # own doors start at the room's floor (98% exactly, else <= 0.2 m steps); storeys are >2 m apart + +SOURCE = "IL3D_3dfront" +IL3D = "imChuling__3D_Room_Collections/IL3D_exported" +F3D = "BillLin66__3D_Room_Collections/3DFront_exported/3dfront.jsonl" +HOUSES = "BillLin66__3D_Room_Collections/3DFront_exported/data/3D-FRONT" + + +def load_assets(root): + with open(os.path.join(root, IL3D, "IL3D/assets.json")) as f: + return {a["model_id"]: a for a in json.load(f)} + + +def placed_size(f, asset): + """Placed extents along mesh x, z, y = IR local X, Y, Z (before the front offset); None if unknown.""" + m = (asset or {}).get("meta_data") or {} + s = f["source_fields"]["layout_object"].get("scale") + if not (s and len(s) == 3 and all(isinstance(m.get(k), (int, float)) for k in ("width", "length", "height"))): + return None + return {"width": m["width"] * abs(s[0]), "length": m["height"] * abs(s[2]), "height": m["length"] * abs(s[1])} + + +def flag_anchor(f): + fl = {k for k, v in (f.get("placement_flags") or {}).items() if v} + z = (f.get("furniture_position") or {}).get("z") + if z is None or len(fl) != 1: + return None + if fl == {"on_floor"} and z < 0.05: + return "floor" + if fl == {"on_ceiling"} and z > 1.0: + return "ceiling" + if fl == {"on_wall"} and z >= 0.05: + return "wall" + return None + + +def il3d_extra(f): + if f.get("structure"): + return {"structure": True, **({"listed_on": f["listed_on"]} if f.get("listed_on") else {})} + return {"anchor": flag_anchor(f), "desc": f.get("description"), + **({"tilt_deg": f["tilt_deg"]} if "tilt_deg" in f else {})} + + +def untilt(f): + """Tilted placement (rotation x/y null) -> the same dict as an upright box (module doc for the matrix): a local axis + within internscenes.TILT_TOL_DEG of vertical -> the exact box with permuted sizes, rotation {0, 0, yaw}; otherwise + the yaw-aligned bounding box of its 8 corners, rotation {None, None, yaw} (convert_room flags it tilted) and + tilt_deg. The box centre is kept; position = the new box's bottom centre.""" + r, p, s = f.get("furniture_rotation") or {}, f.get("furniture_position"), f.get("furniture_size") + R = f["source_fields"].get("rotation_matrix_zup") + if (r.get("x") == 0 and r.get("y") == 0) or not (p and all(_num(p.get(k)) for k in "xyz") and s and R): + return f + R, S = np.asarray(R, dtype=float), [s["width"], s["length"], s["height"]] + c = np.array([p["x"], p["y"], p["z"]]) + R[:, 2] * S[2] / 2 + up = upright(R, S) + if up: + (w, l, h), yaw, _ = up + rot, extra = {"x": 0.0, "y": 0.0, "z": yaw}, {} + else: + (w, l, h), yaw, _, tilt = tilted_bbox(R, S) + rot, extra = {"x": None, "y": None, "z": yaw}, {"tilt_deg": tilt} + return {**f, "furniture_position": {"x": float(c[0]), "y": float(c[1]), "z": float(c[2]) - h / 2}, + "furniture_rotation": rot, "furniture_size": {"width": w, "length": l, "height": h}, **extra} + + +def export_rooms(root): + """house uid -> [(source room id, hull in house frame, height, [(jid, x, y), ...], (doors, windows))].""" + out = collections.defaultdict(list) + for line in open(os.path.join(root, F3D)): + d = json.loads(line) + r = d["room"] + pts = [(f["source_fields"]["jid"], f["furniture_position"]["x"], f["furniture_position"]["y"]) + for f in r["furniture"] if f["source_fields"].get("jid") and f["furniture_position"]] + out[d["house_uid"]].append((d["provenance"]["source_room_id"], r["room_boundary"], r["room_height"], pts, + (r.get("doors") or [], r.get("windows") or []))) + return out + + +def floor_polygon(tris): + """Floor triangles [(x, y) x3] -> the exact floor ring [[x, y], ...], or None unless it is one hole-free polygon.""" + u = unary_union([t for t in map(Polygon, tris) if t.area > 0]) + e = SPIKE_M / 2 # mitre closing then opening: exact except for features narrower than SPIKE_M + u = u.buffer(e, join_style="mitre").buffer(-2 * e, join_style="mitre").buffer(e, join_style="mitre") + parts = [g for g in getattr(u, "geoms", [u]) if g.area >= ARTEFACT_M2] + if len(parts) != 1 or any(Polygon(r).area >= ARTEFACT_M2 for r in parts[0].interiors): + return None + ring = [[round(x, 6), round(y, 6)] for x, y in Polygon(parts[0].exterior).simplify(1e-4).exterior.coords[:-1]] + return ring if len(ring) >= 3 and Polygon(ring).is_valid else None + + +def cm_valid(b): + """The boundary as the model sees it (convert_room's min-corner frame, then scene.clean_boundary) is valid.""" + x0, y0 = min(p[0] for p in b), min(p[1] for p in b) + return clean_boundary([[x - x0, y - y0] for x, y in b]) is not None + + +def floor_polygons(path): + """Raw 3D-FRONT house JSON -> {source room id (instanceid): (floor_polygon of its Floor meshes, floor z)}, house + frame; (None, None) without Floor meshes.""" + with open(path) as f: + h = json.load(f) + mesh = {m.get("uid"): m for m in h.get("mesh") or []} + out = {} + for r in (h.get("scene") or {}).get("room") or []: + tris, zs = [], [] + for c in r.get("children") or []: + m = mesh.get(c.get("ref")) + if m and m.get("type") == "Floor": # mesh children carry identity transforms (as the export assumes) + v, fc = m["xyz"], m["faces"] + p = [(v[i], -v[i + 2]) for i in range(0, len(v) - 2, 3)] + tris += [(p[fc[i]], p[fc[i + 1]], p[fc[i + 2]]) for i in range(0, len(fc) - 2, 3)] + zs += v[1::3] + out[r.get("instanceid")] = (floor_polygon(tris) if tris else None, min(zs) if zs else None) + return out + + +def on_edge(door, ring, floor_z): + """True if the door starts at this room's floor level (not another storey of the house) and its footprint runs + along the room's floor edge for >= half its width, i.e. it opens into the room. On 1,571 sampled rooms 877 of 881 + own doors pass; other rooms' doors cover either >= 0.9 of their width (the far side of the same wall) or <= 0.2 + (a corner touch).""" + pr, p, h = [q[:2] for q in door.get("profile") or []], door.get("position") or {}, door.get("height") + if len(pr) < 3 or not (_num(door.get("width")) and _num(p.get("z")) and _num(h) and _num(floor_z)): + return False + if abs(p["z"] - h / 2 - floor_z) > LEVEL_TOL: + return False + fp = Polygon(pr).buffer(DOOR_TOL, join_style="mitre") + return LinearRing(ring).intersection(fp).length >= 0.5 * door["width"] + + +def openings(doors_windows, shift, prefix): + """Export doors / windows of one source room -> (structure furniture dicts in IL3D's frame, n_skipped). + rotation z = wall direction + 90 so that convert_room(front_offset_deg=-90) gives yaw = wall direction and + size [width, depth, height].""" + out, skipped = [], 0 + for kind, items in zip(("door", "window"), doors_windows): + boxes = [] + for k, o in enumerate(items): + p, w, h, pr = o.get("position"), o.get("width"), o.get("height"), o.get("profile") or [] + xy = [q[:2] for q in pr] + segs = [(a, b) for a, b in zip(xy, xy[1:] + xy[:1]) if math.dist(a, b) > 1e-3] + if not (p and all(_num(p.get(c)) for c in "xyz") and _num(w) and _num(h) and w > 0 and h > 0 and segs): + skipped += 1 + continue + a, b = min(segs, key=lambda s: abs(math.dist(*s) - w)) # the side along the wall + depth = Polygon(xy).area / w if len(pr[0]) == 2 and len(xy) >= 3 else 0.0 + z = o["sill_height"] if _num(o.get("sill_height")) else p["z"] - h / 2 + if h < MIN_H: + skipped += 1 + continue + ang = math.atan2(b[1] - a[1], b[0] - a[0]) + d = depth if depth > 0.01 else OPENING_T + fp = Polygon(footprint({"pos": [p["x"], p["y"]], "size": [w, d], "yaw": ang})) + boxes.append((fp.area * h, fp, { + "furniture_category": kind, "furniture_instance_id": f"{prefix}::{kind}_{k}", "structure": True, + **({"listed_on": o["listed_on"]} if o.get("listed_on") else {}), + "furniture_position": {"x": p["x"] - shift[0], "y": p["y"] - shift[1], "z": z}, + "furniture_rotation": {"x": 0.0, "y": 0.0, "z": math.degrees(ang) + 90}, + "furniture_size": {"width": d, "length": w, "height": h}})) + kept = [] # biggest piece first; a piece half inside the kept ones is the same opening + for _, fp, f in sorted(boxes, key=lambda t: -t[0]): + if kept and unary_union([g for g, _ in kept]).intersection(fp).area >= 0.5 * fp.area: + skipped += 1 + continue + kept.append((fp, f)) + out += [f for _, f in kept] + return out, skipped + + +def frame_shift(furniture, pts, tol=0.02): + """Shift s with export_pos = il3d_pos + s supported by the most shared placements; (s, votes) or (None, 0).""" + pairs = [((f["furniture_position"]["x"], f["furniture_position"]["y"]), pts[f["furniture_asset_id"]]) + for f in furniture if f.get("furniture_position") and f.get("furniture_asset_id") in pts] + best, votes = None, 0 + for p, cands in pairs: + for q in cands: + s = (q[0] - p[0], q[1] - p[1]) + n = sum(any(abs(c[0] - a[0] - s[0]) < tol and abs(c[1] - a[1] - s[1]) < tol for c in cs) for a, cs in pairs) + if n > votes: + best, votes = s, n + return best, votes + + +def split_by_source_room(furniture, rooms, own=None, tol=0.02): + """-> (shift, {export room index: [furniture]}, [unassigned furniture]).""" + pts = collections.defaultdict(list) + for k, r in enumerate(rooms): + for jid, x, y in r[3]: + pts[jid].append((x, y, k)) + shift, _ = frame_shift(furniture, {j: [(x, y) for x, y, _ in v] for j, v in pts.items()}, tol) + hits = [] + for f in furniture: + p = f.get("furniture_position") + hits.append({k for x, y, k in pts.get(f.get("furniture_asset_id"), ()) + if shift and p and abs(x - p["x"] - shift[0]) < tol and abs(y - p["y"] - shift[1]) < tol}) + # a placement listed in several source rooms goes to the room most of this room's objects come from, + # ties to IL3D's own room + votes = collections.Counter(k for h in hits for k in h) + for k, r in enumerate(rooms): + votes[k] += 0.5 * (r[0] == own) + parts, rest = collections.defaultdict(list), [] + for f, h in zip(furniture, hits): + (parts[max(sorted(h), key=votes.__getitem__)] if h else rest).append(f) + return shift, parts, rest + + +def load(root): + assets = load_assets(root) + exp = export_rooms(root) + floors = {} + for scene in iter_records(os.path.join(root, IL3D, "3dfront.json")): + for room in scene["rooms"]: + sf = room["source_fields"] + house = sf["original_house_file"][:-len(".json")] + if house not in floors: + path = os.path.join(root, HOUSES, house + ".json") + floors[house] = floor_polygons(path) if os.path.exists(path) else {} + furniture = [] + for f in room["furniture"]: + f = dict(f) + a = assets.get(f.get("furniture_asset_id")) + # 3D-FUTURE category; for the ~12k assets without one (IL3D label "Others": potted plants, fridges, + # pianos ...) the asset's own assets.json meta_data category + f["furniture_category"] = (f.get("asset_category") or ((a or {}).get("meta_data") or {}).get("category") + or f.get("furniture_category")) + f["furniture_size"] = placed_size(f, a) + furniture.append(f) + rooms = exp.get(house, []) + shift, parts, rest = split_by_source_room(furniture, rooms, sf.get("original_roomId")) + if len(parts) == 1: # unassigned objects (NaN position) can only belong to the one source room + parts = {k: v + rest for k, v in parts.items()} + for k, fl in (parts or {None: furniture}).items(): + srid, hull, height = rooms[k][:3] if k is not None else (None, None, None) + if height is None and srid == sf.get("original_roomId"): + height = room.get("room_height") + poly, floor_z = floors[house].get(srid, (None, None)) + if poly and not cm_valid([[x - shift[0], y - shift[1]] for x, y in poly]): + poly = None + ring = poly or hull + boundary = [[x - shift[0], y - shift[1]] for x, y in ring] if ring else None + uid = f"il3d:{scene['scene_id']}" + (f"::{srid}" if len(parts) > 1 else "") + ops, n_skip = [], 0 + if k is not None and len(rooms[k]) > 4: + doors, windows = rooms[k][4] + doors = doors + [{**o, "listed_on": r[0]} for j, r in enumerate(rooms) if j != k and ring + for o in r[4][0] if on_edge(o, ring, floor_z)] + ops, n_skip = openings((doors, windows), shift, uid) + ir = convert_room({**room, "furniture": [untilt(f) for f in fl] + ops, "room_boundary": boundary, + "room_height": height}, + source=SOURCE, uid=uid, group=f"3dfront:{house.lower()}", subset="3D-FRONT", + boundary_type="polygon" if poly else "hull", front_offset_deg=-90, extra=il3d_extra, + meta={"house": house, "source_room_id": srid, "il3d_room_id": sf.get("original_roomId"), + "il3d_scene_id": scene["scene_id"], "split_from_il3d_room": len(parts) > 1, + "boundary_from": "3dfront_floor_mesh_polygon" if poly else "3dfront_export_floor_hull", + "front_known": True, "n_openings": len(ops), "n_openings_skipped": n_skip, + "n_doors_listed_on_neighbour": sum("listed_on" in o for o in ops), + "size_from": "assets.json mesh extents x layout scale"}) + if len(parts) > 1 and rest: # objects whose source room is unknown: the split rooms are incomplete + ir["meta"]["n_incomplete"] += len(rest) + yield ir + + +if __name__ == "__main__": + f = {"furniture_position": {"x": 1.0, "y": 2.0, "z": 0.0}, "furniture_asset_id": "a", + "placement_flags": {"on_floor": True}, "source_fields": {"layout_object": {"scale": [-2, 1, 1]}}} + assert placed_size(f, {"meta_data": {"width": 1, "length": 0.5, "height": 3}}) == {"width": 2, "length": 3, "height": 0.5} + assert flag_anchor(f) == "floor" + g = {**f, "furniture_position": {"x": 3.0, "y": 2.0, "z": 0.0}, "furniture_asset_id": "b"} + h = {**f, "furniture_position": None} + assert frame_shift([f, g], {"a": [(6.0, 7.0), (0.0, 0.0)], "b": [(8.0, 7.0)]}) == ((5.0, 5.0), 2) + shift, parts, rest = split_by_source_room([f, g, h], [("r0", None, None, [("a", 6.0, 7.0)]), + ("r1", None, None, [("b", 8.0, 7.0), ("a", 0.0, 0.0)])]) + assert shift == (5.0, 5.0) and parts == {0: [f], 1: [g]} and rest == [h] + # a door on a wall running +Y (footprint 0.9 along the wall, 0.24 deep) and a window on a wall running +X + door = {"position": {"x": 6.0, "y": 7.45, "z": 1.05}, "width": 0.9, "height": 2.1, "sill_height": None, + "profile": [[5.88, 7.0], [5.88, 7.9], [6.12, 7.9], [6.12, 7.0]]} + win = {"position": {"x": 8.0, "y": 5.0, "z": 1.65}, "width": 1.2, "height": 1.5, "sill_height": 0.9, + "profile": [[7.4, 5.0, 0.9], [7.4, 5.0, 2.4], [8.6, 5.0, 2.4], [8.6, 5.0, 0.9]]} + pane = {**win, "profile": [[7.5, 5.0, 0.9], [7.5, 5.0, 2.4], [8.5, 5.0, 2.4], [8.5, 5.0, 0.9]], "width": 1.0} + sill = {**win, "height": 0.05} + ops, skip = openings(([door], [pane, win, sill, {"position": None}]), (5.0, 5.0), "t") + assert skip == 3 and [o["furniture_category"] for o in ops] == ["door", "window"], ops # pane merged, sill dropped + assert ops[1]["furniture_size"]["length"] == 1.2 + ir = convert_room({"furniture": ops, "room_boundary": [[0, 0], [4, 0], [4, 3], [0, 3]]}, source="t", uid="t", group="g", + boundary_type="hull", front_offset_deg=-90, extra=il3d_extra) + d, w = ir["objects"] + assert d["structure"] and abs(d["size"][0] - 0.9) < 1e-9 and abs(d["size"][1] - 0.24) < 1e-6 and d["size"][2] == 2.1, d + assert abs(d["pos"][0] - 1) < 1e-9 and abs(d["pos"][1] - 2.45) < 1e-9 and abs(d["pos"][2]) < 1e-9, d + assert abs(d["yaw"] - math.pi / 2) < 1e-9 and w["size"] == [1.2, OPENING_T, 1.5] and w["pos"][2] == 0.9 and abs(w["yaw"]) < 1e-9, w + # the same door on the neighbour's list is merged into the own one; a neighbour-only door keeps its provenance + nb = {**door, "position": {**door["position"], "y": 8.45}, "profile": [[5.88, 8.0], [5.88, 8.9], [6.12, 8.9], [6.12, 8.0]]} + ops, skip = openings(([door, {**door, "listed_on": "r1"}, {**nb, "listed_on": "r1"}], []), (5.0, 5.0), "t") + assert skip == 1 and [o.get("listed_on") for o in ops] == [None, "r1"], ops + ir = convert_room({"furniture": ops, "room_boundary": [[0, 0], [4, 0], [4, 3], [0, 3]]}, source="t", uid="t", group="g", + boundary_type="polygon", front_offset_deg=-90, extra=il3d_extra) + assert "listed_on" not in ir["objects"][0] and ir["objects"][1]["listed_on"] == "r1" + # exact floor: an L from two meshes with a 0.05 mm seam and a sliver -> one 6-vertex ring; a real hole or two + # separate floors -> None (the hull is kept) + rect = lambda x0, y0, x1, y1: [((x0, y0), (x1, y0), (x1, y1)), ((x0, y0), (x1, y1), (x0, y1))] + ring = floor_polygon(rect(0, 0, 4, 2) + rect(0, 2.00005, 2, 4) + [((5, 5), (5.01, 5), (5, 5.01))]) + assert len(ring) == 6 and abs(Polygon(ring).area - 12) < 1e-3 and Polygon(ring).is_valid, ring + # a 5 mm slit between two meshes and a 12 cm zero-width spike (both collapse on the cm grid) -> the 4 x 3 room + ring = floor_polygon(rect(0, 0, 2, 3) + rect(2.005, 0, 4, 3) + [((4, 1), (4.12, 1.0005), (4, 1.001))]) + assert len(ring) == 4 and abs(Polygon(ring).area - 12) < 1e-6, ring + assert floor_polygon(rect(0, 0, 3, 1) + rect(0, 2, 3, 3) + rect(0, 1, 1, 2) + rect(2, 1, 3, 2)) is None + assert floor_polygon(rect(0, 0, 1, 1) + rect(2, 0, 3, 1)) is None + # room il3d:b68c68fc: a vertex lands on another edge once rounded to cm in the room frame + assert cm_valid(ring) and not cm_valid([[5.076, 3.18], [5.076, 0], [6, 0], [6, 4], [5.12, 4], [5.12, 3.14], [5, 3.26], [5, 3.18]]) + # a door on the far side of this room's wall opens into it; a door that only touches its corner does not + room, dz = [[0, 0], [4, 0], [4, 3], [0, 3]], {"position": {"x": 0, "y": 0, "z": 1.05}, "height": 2.1} + assert on_edge({**dz, "width": 0.9, "profile": [[1, 3], [1.9, 3], [1.9, 3.24], [1, 3.24]]}, room, 0.0) + assert not on_edge({**dz, "width": 0.9, "profile": [[1, 3], [1.9, 3], [1.9, 3.24], [1, 3.24]]}, room, -2.9) # upstairs + assert not on_edge({**dz, "width": 0.9, "profile": [[4, 3], [4.24, 3], [4.24, 3.9], [4, 3.9]]}, room, 0.0) + # tilted placement (Z-up matrix, 30 deg about local X) -> yaw-aligned bounding box, flagged tilted; lying flat + # (90 deg about X: local Y vertical) -> the exact box with permuted sizes, not tilted; yaw-only -> unchanged + rx = lambda a: [[1, 0, 0], [0, math.cos(a), -math.sin(a)], [0, math.sin(a), math.cos(a)]] + t = {"furniture_category": "vase", "furniture_position": {"x": 1.0, "y": 1.0, "z": 0.5}, + "furniture_rotation": {"x": None, "y": None, "z": 0.0}, "furniture_size": {"width": 1.0, "length": 0.2, "height": 0.1}, + "source_fields": {"rotation_matrix_zup": rx(math.pi / 6)}} + c30, s30 = math.cos(math.pi / 6), math.sin(math.pi / 6) + lying = untilt({**t, "source_fields": {"rotation_matrix_zup": rx(math.pi / 2)}}) + assert untilt({**t, "furniture_rotation": {"x": 0.0, "y": 0.0, "z": 0.0}})["furniture_position"]["z"] == 0.5 + ir = convert_room({"furniture": [untilt(t), lying], "room_boundary": [[0, 0], [3, 0], [3, 3], [0, 3]]}, source="t", + uid="t", group="g", front_offset_deg=-90, extra=il3d_extra) + o, q = ir["objects"] + close = lambda a, b: all(abs(x - y) < 1e-9 for x, y in zip(a, b)) + assert o["tilted"] and abs(o["tilt_deg"] - 30) < 1e-9 and close(o["size"], [0.2 * c30 + 0.1 * s30, 1.0, 0.2 * s30 + 0.1 * c30]), o + assert close(o["pos"], [1, 1 - 0.05 * s30, 0.5 - 0.1 * s30]) and abs(o["yaw"] - 1.5 * math.pi) < 1e-9, o # centre kept + assert not q["tilted"] and "tilt_deg" not in q and close(q["size"], [0.1, 1.0, 0.2]) and close(q["pos"], [1, 0.95, 0.4]), q + print("il3d_3dfront self-check ok") diff --git a/code/fastfill/adapters/il3d_synthetic.py b/code/fastfill/adapters/il3d_synthetic.py new file mode 100644 index 0000000000000000000000000000000000000000..38c3a8dad4eaffc100f2391935c1eb17b6266e82 --- /dev/null +++ b/code/fastfill/adapters/il3d_synthetic.py @@ -0,0 +1,150 @@ +"""IL3D "Synthetic Data" subset: 6,954 generated single rooms furnished with HSSD assets (IL3D's own supplement). + +Conventions (checked against IL3D's USDZ meshes, see il3d_3dfront.py for the shared parts): +- size = assets.json mesh extents x |layout scale| (the layout bbox is the unscaled mesh extent). +- position = the asset origin, which for HSSD meshes is NOT always the bottom center (hanging lamps, wall shelves). + The mesh bounds come from the HSSD-200 asset catalog (XXXpilar__hssd_clean/asset_catalog.csv, same GLB frame as + IL3D's USDZ, checked on 79 assets) plus EXTRA_BOUNDS for the 54 assets it lacks. +- boundary = the generated rectangular floor (exact for these rooms); no room height in the source. +- category = the asset's category (furniture_category = asset_category = layout category for all 47,837 objects); the + box is the asset mesh the generator substituted, so its category is what the box is (a "bed" slot filled with a + 3.0 x 0.11 x 1.18 m railing mesh stays a railing). The layout's own label (object_name, e.g. "dryer-0") is kept + as layout_name. Only when the asset category is the generic "furniture" (988 objects, HSSD label "Others": washers, + dryers, dishwashers, shelving) the layout label names the object better, so it becomes the category (instance + suffix dropped) and the asset category is kept as asset_category. Of those, 1 ("floor_drain_cover" on a washer + mesh) is thereby shown as a fixed structure box instead of placed. Against the mesh's HSSD synset (catalog + wnsynsetkey) the name agrees for 449, differs for 453 and the mesh has none for 86; the differences are mostly + look-alike swaps ("dryer" on 209 washing-machine placements, "washing_machine"/"dryer" on 78 of a "DISHWASHER HOBART" + mesh, shelving names on a bathroom rack filed as handcart.n.01). The name is used anyway: it is the layout's label + of the slot whose pose is the target, the size already is the mesh's, and a name-agrees-with-synset rule would put + 539 back to "furniture". +""" +import csv +import json +import math +import os +import re + +from fastfill.adapters.il3d_3dfront import IL3D, il3d_extra, load_assets, placed_size +from fastfill.adapters.unified import convert_room, iter_records + +SOURCE = "IL3D_synthetic" +FRONT_OFFSET_DEG = -90 # HSSD mesh +Z -> IR local -Y (front test) +GENERIC_ASSET = "furniture" # an asset category that names nothing: the layout's object_name is used instead + +# mesh bounds (m, y-up asset frame) of the 54 IL3D HSSD assets absent from the HSSD-200 catalog, read from +# IL3D HSSD.zip USDZ (extents match assets.json meta for all 54) +EXTRA_BOUNDS = { + "08d386c25702f0613120b54e2b0615b4e4d198f7": ([-0.828425, 0.000115, -0.215669], [0.828425, 1.464155, 0.215669]), + "0a3e0a10ec97c41a36939e89b0591d2cfe5e41e1": ([-0.6, -0.0, -0.651672], [0.6, 0.882244, 0.651672]), + "103a70c6e8d54ac0cfc0fe28d66a8bea6b7c9144": ([-0.762062, -0.000101, -0.766263], [0.762062, 0.939839, 0.766263]), + "1b583238d6351379093ac7ac1df38699568bb4ec": ([-0.60996, -0.0, -0.2159], [0.609961, 0.787413, 0.2159]), + "1cdcc88935c2babba66677fa1d95aee0e6a410f3": ([-0.395, -0.0, -0.165], [0.395, 0.7, 0.165]), + "1e4e58bf53e51df27beeb774b5d70818de124068": ([-0.323064, 0.0, -0.397458], [0.323064, 0.924784, 0.397458]), + "252ea1f9106ada162cf68bb61c71efd4a62d4c87": ([-0.701118, -0.0, -0.16], [0.701119, 1.474944, 0.16]), + "28f1e0bae95e70346996067a45e4e32c2e1b7d75": ([-1.106139, -0.002024, -0.792466], [1.106139, 0.854505, 0.792466]), + "2a6fa282275e6ddb1fc35ebdc6a9ef91ae93a8ae": ([-0.45, 1.21079, -0.153132], [0.45, 2.274057, 0.153132]), + "31a2b3d52aafa438f763b1d4a6b60297b95fa4e4": ([-0.250502, -3e-06, -0.089994], [0.250502, 0.161684, 0.089994]), + "36b072f5c0db91a6b77fb0bfeaef686ee23a37f9": ([-1.338968, -0.008765, -1.339009], [1.339019, 2.507386, 1.338978]), + "381f06fc43fe78e832a882769242e022dea8c81e": ([-0.751624, -0.000244, -0.799559], [0.747077, 1.023986, 0.798994]), + "38da5efc3b4c3c250718e555cf68439746c71c65": ([-0.8, -0.0, -0.578804], [0.8, 0.75, 0.578804]), + "3a886310f67f7cca9d6ea34b5b1b28bc43f282f5": ([-0.69831, 0.0, -0.521203], [0.69831, 1.040641, 0.521203]), + "3a8afefa6b4a735293520f5bd14f3ec9d456a004": ([-1.82, -0.0, -0.251171], [1.819989, 1.97001, 0.251171]), + "3dca32a89b68041131bf3b0db5a07b80ad26c0c7": ([-0.53, 0.0, -0.225], [0.53, 1.9, 0.225]), + "3e8fb8993fed7051cb6da2ee1001d1cba046aaf8": ([-0.466968, -6.3e-05, -0.629036], [0.659181, 0.798143, 0.629036]), + "46e08ada1af32862e55de18a583e8f60e7fa7917": ([-0.803414, -0.00243, -0.299899], [0.713549, 0.850902, 0.30151]), + "47a95d3baa393487ba90ce9c90509cff0c9bb4e6": ([-0.43042, -0.0, -0.1524], [0.430419, 1.067658, 0.1524]), + "4d36e3731f127abfb07f1d970239d6d9f21e1b3e": ([-0.703974, -0.0, -0.173289], [0.703974, 0.741946, 0.17329]), + "4d6310a9ffa5b50e7dc45940cf724ba1fa3ffddd": ([-1.850866, -0.015017, -0.288937], [1.850866, 2.696201, 0.288936]), + "51824f2ecd931fbed1389e715e8184354c6f62d4": ([-0.55, -0.004554, -0.127143], [0.55, 0.452, 0.065389]), + "520a96262a7843c84a8cbebc172e70830d71c2db": ([-0.185151, -0.0, -0.158055], [0.185151, 1.602131, 0.158055]), + "5be064296d6117e1151e58b8f858b9147b6279a9": ([-0.64419, -9e-05, -0.175091], [0.644901, 1.667616, 0.175091]), + "605f233dd70c9a66d02914f1420d964642bdba81": ([-1.07618, -0.0, -1.382074], [1.07618, 0.836965, 1.382071]), + "62fd5aea80235288d1896ddab73cdcef97a6a585": ([-0.630063, -0.0, -0.536025], [0.630063, 0.859829, 0.536025]), + "6cf17da9a4062b73cafe548996b92e167d023a9a": ([-0.562498, -0.0, -0.510002], [0.562497, 0.9, 0.510002]), + "7718968f80499672922d0434c8c617b0902f0bad": ([-0.810003, 0.0, -0.6125], [0.810003, 0.717001, 0.612501]), + "777d1748a60d9b3ae52ad799aa0e215fd22715c6": ([-0.461928, -1.1e-05, -0.207056], [0.452472, 0.7493, 0.199345]), + "799439ef082a442613ddcadd2603aeeeb477b13e": ([-0.7, 0.00325, -0.7], [0.7, 0.45887, 0.7]), + "79d4fe8642358909adaa07a8b05793bcdd7f4450": ([-0.701118, -0.0, -0.16], [0.701119, 1.474944, 0.16]), + "8597e0af462124aef9ec06dde13f45bb2b223a74": ([-0.5, -7.7e-05, -0.614043], [0.5, 1.09571, 0.611347]), + "88ed586afa70ba6619fb122f205253a4a9d74aaa": ([-0.305, -0.0, -0.165], [0.305, 1.830585, 0.165]), + "92bcbbdf57f3668c4b6192b9de9dfd95640de0d1": ([-0.95, 0.000194, -0.225], [0.95, 1.060194, 0.225001]), + "9a1c224cd11be35a5c28347de7868674f56789da": ([-0.15, -0.002598, -0.45], [0.15, 0.3, 0.45]), + "a375e44843d9ca2e2dda1bae8b65edbc8b84bbf0": ([-0.776006, 3.5e-05, -0.775123], [0.775905, 0.978471, 0.775655]), + "a59f443c419c8514ecface071b4fd67bd44b4471": ([-0.415, -0.0, -0.1925], [0.415, 1.45, 0.192501]), + "a60150783b79e0fb4cb3d938483f718585c52bda": ([-0.51, -0.0, -0.215], [0.51, 1.83, 0.215]), + "a7323daeac0ded5ebafdb6231be3932e88a5b7f3": ([-0.45085, -0.0, -0.333766], [0.45085, 1.638814, 0.333766]), + "b827ad39b820d4908a4222dc95dd4b63ecf6c0a6": ([-0.45, 1.21079, -0.153132], [0.45, 2.274053, 0.153132]), + "ba28803b05660ca87ad0650276988f02dce1081e": ([-0.774645, -0.0, -0.774677], [0.774645, 0.914617, 0.774677]), + "bc58cc78e7268f25da1dbee0f989c25c7e3d1108": ([-1.82, -0.0, -0.2975], [1.820001, 1.898, 0.2975]), + "bf70431f51b8e06f2d7ef70916682dcd943553b8": ([-0.71163, -0.0, -0.580984], [0.716089, 0.99339, 0.3]), + "c184f9647c5788e29d55dbf622bae24513ec48ec": ([-0.1775, -0.0, -0.145], [0.1775, 1.310004, 0.145]), + "c4492609d027c557c5dc1ab8fc83bdd2e7617e12": ([-0.449618, -0.0, -0.183844], [0.449618, 1.52999, 0.150187]), + "ca6bc7678efa7a97f59ee6a650217a5492b3a0f8": ([-0.770001, -0.0, -0.764999], [0.770001, 0.7, 0.764999]), + "cee999e8b57589a4862425c588838f28bcc56f3b": ([-0.702015, -1e-06, -1.076095], [0.702015, 0.890897, 1.076095]), + "d724fb718897f8200e6203a094369dc5e6c45ca2": ([-0.550002, 0.0, -0.45], [0.550002, 0.77, 0.45]), + "dd56f602c6c29a97bbff71f4e48f07454ec97b62": ([-0.521833, 0.001956, -0.773037], [0.521833, 0.385286, 0.773038]), + "efb3a025089c14264fde9b03c67d7353fb6759c5": ([-0.5334, -0.0, -0.1524], [0.5334, 1.57988, 0.167796]), + "f8261de437a1f1f23263e4df0deb0a493ed52387": ([-0.530916, -0.0, -0.3048], [0.530916, 0.662856, 0.3048]), + "ff0d0a7fe6e2a685cf03bf6ef2e32ae44d79372d": ([-0.45, 1.21079, -0.153132], [0.45, 2.274057, 0.153132]), + "ff587a04fe3412b83b8fc870a7216374e70506c4": ([-0.828425, 0.000114, -0.215669], [0.828425, 1.473051, 0.215669]), + "xxxxbc6e2459xae45x49d0x86c4xe27c2b0767a7": ([-0.399916, 0.0, -0.2032], [0.399916, 1.942959, 0.2032]), +} + + +def load_catalog(root): + with open(os.path.join(root, "XXXpilar__hssd_clean/asset_catalog.csv")) as f: + return {r["template_name"]: (json.loads(r["local_bbox_min"]), json.loads(r["local_bbox_max"])) + for r in csv.DictReader(f) if r["local_bbox_min"]} | EXTRA_BOUNDS + + +def bottom_center(f, bounds): + """Asset-origin position -> bottom center of the placed box (IR frame), or None.""" + p, rot = f.get("furniture_position"), f.get("furniture_rotation") or {} + if not (p and bounds and rot.get("z") is not None): + return None + (x0, y0, z0), (x1, y1, z1) = bounds + s = f["source_fields"]["layout_object"]["scale"] + cx, cz, by = s[0] * (x0 + x1) / 2, s[2] * (z0 + z1) / 2, min(s[1] * y0, s[1] * y1) + yaw = math.radians(rot["z"]) # yaw of mesh +X; mesh +Z is IR local -Y + c, n = math.cos(yaw), math.sin(yaw) + return {"x": p["x"] + c * cx + n * cz, "y": p["y"] + n * cx - c * cz, "z": p["z"] + by} + + +def labels(f): + """-> (IR category, extra IR fields): the layout label is always kept; it is the category only for GENERIC_ASSET.""" + cat, name = f.get("furniture_category"), f.get("object_name") + if cat == GENERIC_ASSET and name: + return re.sub(r"-\d+$", "", name), {"layout_name": name, "asset_category": cat} + return cat, {"layout_name": name} if name else {} + + +def load(root): + assets, catalog = load_assets(root), load_catalog(root) + for scene in iter_records(os.path.join(root, IL3D, "synthetic_data.json")): + for room in scene["rooms"]: + furniture = [] + for f in room["furniture"]: + f = dict(f) + f["furniture_size"] = placed_size(f, assets.get(f.get("furniture_asset_id"))) + f["furniture_position"] = bottom_center(f, catalog.get(f.get("furniture_asset_id"))) + f["furniture_category"], f["_labels"] = labels(f) + furniture.append(f) + yield convert_room({**room, "furniture": furniture}, source=SOURCE, uid=f"il3d:{scene['scene_id']}", + group=f"il3d:{scene['scene_id']}", subset="Synthetic Data", boundary_type="polygon", + front_offset_deg=FRONT_OFFSET_DEG, extra=lambda f: {**il3d_extra(f), **f["_labels"]}, + meta={"il3d_scene_id": scene["scene_id"], "front_known": True, + "position_from": "asset origin + HSSD-200 catalog mesh bounds", + "size_from": "assets.json mesh extents x layout scale"}) + + +if __name__ == "__main__": + f = {"furniture_position": {"x": 1.0, "y": 1.0, "z": 2.0}, "furniture_rotation": {"z": 90.0}, + "source_fields": {"layout_object": {"scale": [1, 2, 1]}}} + b = bottom_center(f, ([-0.5, -0.5, 0.0], [0.5, 0.0, 0.4])) # hanging: origin at top, box centered at z=+0.2 + assert all(abs(b[k] - v) < 1e-9 for k, v in {"x": 1.2, "y": 1.0, "z": 1.0}.items()), b + assert labels({"furniture_category": "furniture", "object_name": "clothes_dryer-2"}) == ( + "clothes_dryer", {"layout_name": "clothes_dryer-2", "asset_category": "furniture"}) + assert labels({"furniture_category": "balcony_railing", "object_name": "bed-0"}) == ("balcony_railing", {"layout_name": "bed-0"}) + assert labels({"furniture_category": "furniture"}) == ("furniture", {}) + print("il3d_synthetic self-check ok") diff --git a/code/fastfill/adapters/interiorgs.py b/code/fastfill/adapters/interiorgs.py new file mode 100644 index 0000000000000000000000000000000000000000..2680282212ceaff0f440476a92d13593a58756e9 --- /dev/null +++ b/code/fastfill/adapters/interiorgs.py @@ -0,0 +1,369 @@ +"""InteriorGS (Manycore designer scenes, 1,000 houses / 6,607 rooms). View used: interiorgs.json only (full +scenes, every room once); json_single.json + multiroom/*.json repackage the same rooms and are ignored. + +Upstream room assignment can lose objects at repeated/degenerate profile edges. For scene.unassigned_objects, +recover only a unique room covering the mean of the eight source_fields.bounding_box corners, using the same +conservative ring repair as conversion. Existing assignments stay authoritative. Shared edges, overlapping rooms, +outside centers, invalid geometry and duplicate IDs remain excluded with IDs/reasons in meta.membership_recovery; +no nearest-room assignment. A missing/invalid room polygon prevents proving uniqueness, so recovery is withheld. + +Export (aggregate_scenes.py): meters, Z-up, scene-global XY, room_boundary = structure.rooms[].profile (floor +polygon), room_height from the room_box. The exported furniture_rotation/size are longest-edge geometry +([0, 180) yaw, no front), so this adapter re-derives pose from source_fields.bounding_box instead: +corners 0-3 are the bottom face in a fixed local order and edge c1-c2 is the object's BACK (front test on all +1,000 scenes: bed 0.85, toilet 0.94, painting 0.90 touch the wall with c1-c2, c3-c0 ~0.00), so +front = midpoint(c3, c0) - midpoint(c1, c2); IR size = [|c0c1| (depth, along front), |c1c2|, height]. +Caveat: custom cabinetry (labels wardrobe / cabinet / wall cabinet) does not follow this frame (wall side ~uniform), +so their yaw is only reliable up to the box axes: those objects get front_known=False. +Boxes whose top face is shifted from the bottom face or whose bottom is not level (7,355, all rectangular) go +through internscenes.upright / tilted_bbox as internscenes.canon does, with local X = c0 - c1 (front), Y = c2 - c1, +Z = c4 - c0: 3,704 are upright boxes with permuted axes (corners 0-3 on a side face: strip lights, wall panels, +lying boxes) and 33 lean < 5 deg, re-expressed as yaw-only boxes (yaw = heading of local X); the other 3,618 (cigars +racked at 30 deg, leaning books, storage racks) become the yaw-aligned bounding box of their 8 corners, tilted +(rotation x/y None) with "tilt_deg". Bottom = box centre - height / 2. When local X is the (near-)vertical axis (2,683 +boxes: vents, ceiling mouldings, wall designs, lying food boxes) yaw is the heading of local Y and front_known=False: +the front points up or down, so the source gives no horizontal front. +123 boxes are level but upside down (local Z = c4 - c0 points down: 103 rotated, 20 mirrored; wall designs, vents, +paintings, hanging wine glasses): same path, so height is the real one, not the ~0 of c0-c3 minus themselves, and yaw +stays the heading of c0 - c1 (the world image of local +X, i.e. of the front, whichever way the box was flipped). +Doors / windows: scene["openings"] (= structure.holes, profile = vertical polygon in the wall, scene Z; every room +floor is at z = 0) of type DOOR / WINDOW, matched to a room geometrically, not through room.doors/windows (most +openings have room_indices []): the profile lies on the room's wall line (the ring is the wall's inner face) or on +the wall's centre line or far face, so an opening belongs to every room with a ring edge parallel to it (profile +spread < PAR_TOL off the edge line), between WALL_TOL inside and thickness + WALL_TOL outside it, covering more than +half its width (a door in a shared wall goes to both rooms). Emitted as "structure" boxes on that edge (the wall line): +local X along the wall = the profile's extent along the edge (the export's width = max(X span, Y span) is wrong for +oblique openings), OPENING_T across it (as the other sources' openings; the export's thickness is the wall's), +bottom = min profile z, height = profile z span. 5,714 / 5,716 doors and 6,211 / 6,273 windows match a room (4,554 doors +and 233 windows two); 50 of the unmatched windows are flat chords across curved walls (ring = 0.2 m segments), not +handled. HOLE / OPENING types (passages without a leaf) are not emitted. The export repeats some holes (0293_840751 +holes 15-18 are one window; 0395_840187 holes 23/24 are one door on both faces of a 6 cm wall): a box more than half +inside a bigger one of its kind on the same edge is skipped (32 boxes in 29 rooms: 27 doors, 5 windows; +meta.n_openings_repeated). +Wall rings that are invalid only through a corner overshoot (the ring crosses itself around a few-cm lobe), a zero-area +spur or a self-touching vertex (169 of the 171 invalid rings, area change <= 0.06 m2) are repaired with buffer(0) when +it gives one hole-free polygon within REPAIR_TOL of the ring's shoelace area (meta.boundary_repaired); other invalid +rings get boundary_type None. +""" +import math +import os +import re +from collections import Counter + +import numpy as np +from shapely.geometry import Point, Polygon, box as rect +from shapely.ops import unary_union + +from fastfill.adapters.internscenes import tilted_bbox, upright +from fastfill.adapters.unified import _num, convert_room, iter_records, signed_area +from fastfill.scene import footprint, head + +SOURCE = "InteriorGS" +GROUP = {"0755_840824": "structured3d:scene_01482", # same design as a Structured3D house (room boundary + furniture match) + # duplicated InteriorGS houses that split.py's content_key does not link (rooms differ by a few objects) + "0565_840943": "interiorgs:0478_840944", "0684_841508": "interiorgs:0521_840344"} +TILT_TOL = 0.02 # m +OPENING_T, PAR_TOL, WALL_TOL = 0.10, 0.02, 0.05 # m +REPAIR_TOL = 0.01 # relative area change allowed when repairing a wall ring +# Only the generic custom-cabinetry labels: TV / basin / shoe / wine / display / storage cabinets follow the corner +# order (back edge c1-c2 on the wall 0.51-0.82, front edge c3-c0 <= 0.10), plain wardrobe / cabinet / wall cabinet do not +UNRELIABLE_FRONT = re.compile(r"^(wardrobe|cabinet|wall cabinet)s?( doors)?$", re.I) + + +def _pose(f): + """Export furniture -> unified-schema furniture with front-aware yaw/size, or None fields if no bbox.""" + bb = (f.get("source_fields") or {}).get("bounding_box") + if not bb or len(bb) != 8: + return {**f, "furniture_rotation": None} + c = [(p["x"], p["y"], p["z"]) for p in bb] + bz = [p[2] for p in c[:4]] + if (max(bz) - min(bz) > TILT_TOL or c[4][2] < c[0][2] - TILT_TOL or # bottom not level, or local Z down + max(math.hypot(c[i][0] - c[i + 4][0], c[i][1] - c[i + 4][1]) for i in range(4)) > TILT_TOL): + return _upright(f, np.array(c)) + fx = (c[3][0] + c[0][0] - c[1][0] - c[2][0]) / 2 + fy = (c[3][1] + c[0][1] - c[1][1] - c[2][1]) / 2 + return {**f, + "furniture_position": {"x": sum(p[0] for p in c[:4]) / 4, "y": sum(p[1] for p in c[:4]) / 4, "z": min(bz)}, + "furniture_rotation": {"x": 0.0, "y": 0.0, "z": math.degrees(math.atan2(fy, fx))}, + "furniture_size": {"width": math.hypot(fx, fy), + "length": math.hypot(c[2][0] - c[1][0], c[2][1] - c[1][1]), + "height": max(p[2] for p in c) - min(bz)}} + + +def _upright(f, c): + """8 corners (array 8x3) of a box whose bottom face is not level -> the upright box (permuted axes) or the + yaw-aligned bounding box of the corners (tilted: rotation x/y None, "tilt_deg"), as internscenes.canon.""" + e = np.array([c[0] - c[1], c[2] - c[1], c[4] - c[0]]).T # local X (front), Y, Z edges as columns + size = np.linalg.norm(e, axis=0) + R = e / np.maximum(size, 1e-12) + up = upright(R, size) + if up is None: + (sx, sy, sz), yaw, k, tilt = tilted_bbox(R, size) + rot, extra = None, {"tilt_deg": tilt} + else: + (sx, sy, sz), yaw, k = up + rot, extra = 0.0, {} + x, y, z = c.mean(axis=0) + return {**f, "furniture_position": {"x": x, "y": y, "z": z - sz / 2}, + "furniture_rotation": {"x": rot, "y": rot, "z": yaw}, + "furniture_size": {"width": sx, "length": sy, "height": sz}, "vertical_axis": k, **extra} + + +def openings(scene_openings, ring, prefix): + """Scene DOOR / WINDOW openings in this room's walls -> (structure furniture on the wall line (see module doc), + n_repeated). Rotation z = edge direction, size [extent along the edge, OPENING_T, height] (front_offset_deg 0). + The export repeats some holes (exact profile copies, or the same hole on both faces of a thin wall, a few cm + apart): biggest first, an opening more than half inside the kept ones of its kind on the same edge (in the wall + plane: along-edge x z) is a repeat and is skipped, as il3d_3dfront.openings does.""" + b = [tuple(p[:2]) for p in ring] + if signed_area(b) < 0: + b = b[::-1] + cand = [] + for o in scene_openings: + kind = {"DOOR": "door", "WINDOW": "window"}.get(o.get("opening_type")) + pr, t = o.get("profile") or [], o.get("thickness") + if not (kind and pr and _num(t)): + continue + best = None + for i, ((x0, y0), (x1, y1)) in enumerate(zip(b, b[1:] + b[:1])): + L = math.hypot(x1 - x0, y1 - y0) + if L < 1e-9: + continue + ux, uy = (x1 - x0) / L, (y1 - y0) / L + s = [(p[0] - x0) * ux + (p[1] - y0) * uy for p in pr] # along the edge + n = [(p[1] - y0) * ux - (p[0] - x0) * uy for p in pr] # across it, > 0 inside the (CCW) room + ov = min(max(s), L) - max(min(s), 0) + if max(n) - min(n) < PAR_TOL and min(n) >= -t - WALL_TOL and max(n) <= WALL_TOL and \ + ov > 0.5 * (max(s) - min(s)) and (best is None or ov > best[0]): + best = (ov, i, x0, y0, ux, uy, min(s), max(s)) + if best is None: + continue + _, i, x0, y0, ux, uy, s0, s1 = best + z0, z1 = min(p[2] for p in pr), max(p[2] for p in pr) + cand.append((kind, i, rect(s0, z0, s1, z1), { + "furniture_category": kind, "furniture_instance_id": f"{prefix}::hole_{o['source_hole_index']}", + "structure": True, + "furniture_position": {"x": x0 + ux * (s0 + s1) / 2, "y": y0 + uy * (s0 + s1) / 2, "z": z0}, + "furniture_rotation": {"x": 0.0, "y": 0.0, "z": math.degrees(math.atan2(uy, ux))}, + "furniture_size": {"width": s1 - s0, "length": OPENING_T, "height": z1 - z0}})) + kept, keep = {}, set() + for kind, i, r, f in sorted(cand, key=lambda t: -t[2].area): + ks = kept.setdefault((kind, i), []) + if not (ks and unary_union(ks).intersection(r).area > 0.5 * r.area): + ks.append(r) + keep.add(id(f)) + out = [f for *_, f in cand if id(f) in keep] # source hole order + return out, len(cand) - len(out) + + +def repair_ring(ring): + """Invalid wall ring -> (buffer(0) exterior, meta) when that is one hole-free polygon within REPAIR_TOL of the + ring's shoelace area (corner overshoot, spur, self-touching vertex); else (ring, {}).""" + if not ring or len(ring) < 3 or Polygon(ring).is_valid: + return ring, {} + a, q = abs(signed_area([p[:2] for p in ring])), Polygon(ring).buffer(0) + if q.geom_type != "Polygon" or q.interiors or not abs(q.area - a) < REPAIR_TOL * a: + return ring, {} + return [list(p) for p in q.exterior.coords[:-1]], {"boundary_repaired": "buffer(0)", + "boundary_area_change_m2": round(q.area - a, 4)} + + +def drop_labelled(objs): + """IR objects minus the opening boxes the source already has as a labelled furniture box: a 'window' / 'door' box + of the same head noun that overlaps the opening's footprint and covers more than half of its wall-plane rectangle + (along the wall x z). 207 windows + 4 doors in 127 rooms: mostly boxes centred on the same wall line 0.13 m deep, + 7 window panes 2-5 cm thin a few cm off the line; a bay-window platform labelled 'window' below the opening does + not count. The source's own box stays (meta.n_openings_dropped).""" + own = [(head(o), Polygon(footprint(o)), o) for o in objs if not o.get("structure") and head(o) in ("door", "window")] + + def covered(o): + fp, ux, uy = Polygon(footprint(o)), math.cos(o["yaw"]), math.sin(o["yaw"]) + c = o["pos"][0] * ux + o["pos"][1] * uy + for h, p, q in own: + if h != o["category"] or not p.intersection(fp).area > 0: + continue + s = [x * ux + y * uy for x, y in p.exterior.coords] + ds = min(max(s), c + o["size"][0] / 2) - max(min(s), c - o["size"][0] / 2) + dz = min(q["pos"][2] + q["size"][2], o["pos"][2] + o["size"][2]) - max(q["pos"][2], o["pos"][2]) + if ds > 0 and dz > 0 and ds * dz > 0.5 * o["size"][0] * o["size"][2]: + return True + return False + return [o for o in objs if not (o.get("structure") and covered(o))] + + +def _extra(f): + if f.get("structure"): + return {"structure": True} + unknown = UNRELIABLE_FRONT.search(f["furniture_category"]) or f.get("vertical_axis") == 0 # front up / down + x = {"front_known": False} if unknown else {} + return {**x, "tilt_deg": f["tilt_deg"]} if "tilt_deg" in f else x + + +def _membership_id(f): + """Scene-local export identity, with the raw instance/index metadata as fallbacks.""" + fields = f.get("source_fields") or {} + for prefix, value in (("", f.get("furniture_instance_id")), ("ins:", fields.get("ins_id")), + ("source_index:", f.get("source_index"))): + if value is not None and value != "": + return prefix + str(value) + return None + + +def _membership_polygon(room): + ring = room.get("room_boundary") + if not isinstance(ring, list) or len(ring) < 3 or not all( + isinstance(p, (list, tuple)) and len(p) >= 2 and all(_num(v) for v in p[:2]) for p in ring): + return None + fixed, _ = repair_ring([p[:2] for p in ring]) + poly = Polygon(fixed) + return poly if poly.is_valid and not poly.is_empty and poly.area > 0 else None + + +def _membership_center(f): + corners = (f.get("source_fields") or {}).get("bounding_box") + if not isinstance(corners, list) or len(corners) != 8 or not all( + isinstance(p, dict) and all(_num(p.get(k)) for k in "xyz") for p in corners): + return None + return Point(*(sum(p[k] / 8 for p in corners) for k in "xy")) + + +def _membership_rooms(scene): + """Return (room, recovery metadata) pairs without modifying the export or inventing room membership.""" + rooms, pending = scene["rooms"], scene.get("unassigned_objects") or [] + if not pending: + return [(r, {}) for r in rooms] + polygons = [_membership_polygon(r) for r in rooms] + assigned = {_membership_id(f) for r in rooms for f in r.get("furniture") or []} + counts = Counter(_membership_id(f) for f in pending) + + def locate(f): + identity = _membership_id(f) + if identity is None: + return None, "missing_id" + if identity in assigned: + return None, "already_assigned" + if counts[identity] > 1: + return None, "duplicate_id" + center = _membership_center(f) + if center is None: + return None, "invalid_bbox" + if any(p is None for p in polygons): + return None, "invalid_boundary" + hits = [i for i, p in enumerate(polygons) if p.covers(center)] + return (hits[0], None) if len(hits) == 1 else (None, "ambiguous" if hits else "outside") + + decisions = [(f, _membership_id(f) or f"unassigned:{i}", *locate(f)) for i, f in enumerate(pending)] + reasons = sorted({reason for _, _, _, reason in decisions if reason and reason != "already_assigned"}) + common = {"method": "unique_bbox_center", "scene_unassigned": len(pending), + "scene_recovered": sum(reason is None for _, _, _, reason in decisions), + "scene_already_assigned": [key for _, key, _, reason in decisions if reason == "already_assigned"], + "scene_unresolved": {reason: [key for _, key, _, why in decisions if why == reason] for reason in reasons}} + return [({**room, "furniture": list(room.get("furniture") or []) + [f for f, _, j, _ in decisions if j == i]}, + {"membership_recovery": {**common, "recovered_ids": [key for _, key, j, _ in decisions if j == i]}}) + for i, room in enumerate(rooms)] + + +def load(root): + path = os.path.join(root, "imChuling__3D_Room_Collections/InteriorGS_exported/interiorgs.json") + for scene in iter_records(path): + sid = scene["scene_id"] + for room, recovery in _membership_rooms(scene): + ring, fix = repair_ring(room["room_boundary"]) + furn = [_pose(f) for f in room["furniture"]] + ops, n_rep = openings(scene.get("openings") or [], ring or [], sid) + furn += ops + ir = convert_room({**room, "room_boundary": ring, "furniture": furn}, + source=SOURCE, uid=f"interiorgs:{sid}::{room['room_id']}", group=GROUP.get(sid, f"interiorgs:{sid}"), + boundary_type="polygon", extra=_extra, + meta={"scene_id": sid, "front_known": True, "front_source": "bbox corner order", + "room_type_available": False, "view": "interiorgs.json", + "n_permuted_axes": sum(f.get("vertical_axis", 2) != 2 for f in furn), + "n_openings_repeated": n_rep, **fix, **recovery}) + objs = drop_labelled(ir["objects"]) + ir["meta"]["n_openings_dropped"] = len(ir["objects"]) - len(objs) + ir["objects"] = objs + b = ir["boundary"] + if b and not Polygon(b).is_valid: # self-intersecting source profile: cannot prove in-bounds + ir["boundary_type"] = None + ir["meta"]["boundary_invalid"] = True + yield ir + + +if __name__ == "__main__": + # bed-like box whose back edge c1-c2 lies on the wall x=0 (clockwise corners): front = +X, depth 0.6, width 2 + f = {"source_fields": {"bounding_box": [{"x": x, "y": y, "z": z} for z in (0.0, 1.0) for x, y in + ((0.6, -1.0), (0.0, -1.0), (0.0, 1.0), (0.6, 1.0))]}} + p = _pose(f) + assert abs(p["furniture_rotation"]["z"]) < 1e-9 and p["furniture_rotation"]["x"] == 0.0 + assert abs(p["furniture_size"]["width"] - 0.6) < 1e-9 and abs(p["furniture_size"]["length"] - 2.0) < 1e-9 + assert p["furniture_position"] == {"x": 0.3, "y": 0.0, "z": 0.0} + + from fastfill.adapters.internscenes import rot_zxy + + def box(ctr, R, size): # export corner order: c0 - c1 = local X, c2 - c1 = local Y, c4 - c0 = local Z + loc = [(sx, sy, sz) for sz in (-1, 1) for sx, sy in ((1, -1), (-1, -1), (-1, 1), (1, 1))] + pts = np.asarray(ctr) + (np.array(loc) * np.asarray(size) / 2) @ np.asarray(R).T + return {"source_fields": {"bounding_box": [dict(zip("xyz", q)) for q in pts.tolist()]}} + close = lambda a, b, tol=1e-6: all(abs(x - y) < tol for x, y in zip(a, b)) + pose = lambda p: ([p["furniture_position"][k] for k in "xyz"], p["furniture_rotation"], + [p["furniture_size"][k] for k in ("width", "length", "height")]) + # book lying flat, its local X (front) vertical: upright box with permuted axes, yaw = heading of local Y + X, Y = np.array([0.0, 0, 1]), np.array([math.cos(0.5), math.sin(0.5), 0]) + R = np.stack([X, Y, np.cross(X, Y)], axis=1) + pos, rot, size = pose(_pose(box([1, 2, 0.8], R, [0.3, 0.2, 0.05]))) + assert rot["x"] == 0.0 and abs(rot["z"] - math.degrees(0.5)) < 1e-6 and close(size, [0.2, 0.05, 0.3]), (rot, size) + assert close(pos, [1, 2, 0.65]) + assert _extra({"furniture_category": "book", **_pose(box([1, 2, 0.8], R, [0.3, 0.2, 0.05]))}) == {"front_known": False} + # level box turned upside down (180 deg about local X: corners 4-7 below 0-3): real height, bottom, front kept + p = _pose(box([1, 2, 1.5], np.diag([1.0, -1, -1]), [0.8, 0.4, 0.6])) + pos, rot, size = pose(p) + assert rot["x"] == 0.0 and abs(rot["z"]) < 1e-6 and close(size, [0.8, 0.4, 0.6]) and close(pos, [1, 2, 1.2]), p + assert _extra({"furniture_category": "wall design", **p}) == {} + # truly tilted (30 deg about local X, yaw 90): yaw-aligned bounding box of the 8 corners, flagged via x/y None + p = _pose(box([0, 0, 1], rot_zxy(math.pi / 2, math.radians(30), 0), [1.0, 0.4, 0.2])) + pos, rot, size = pose(p) + c, s_ = math.cos(math.radians(30)), math.sin(math.radians(30)) + assert rot["x"] is None and abs(rot["z"] - 90) < 1e-6 and abs(p["tilt_deg"] - 30) < 1e-6, (rot, p) + assert close(size, [1.0, 0.4 * c + 0.2 * s_, 0.4 * s_ + 0.2 * c]) and close(pos, [0, 0, 1 - size[2] / 2]) + + # 4 x 3 room at (10, 20), ring given clockwise; wall thickness 0.24 / 0.12 + ring = [[10, 20], [10, 23], [14, 23], [14, 20]] + quad = lambda a, b, z0, z1: [[*a, z1], [*b, z1], [*b, z0], [*a, z0]] + ops = [{"opening_type": "DOOR", "source_hole_index": 0, "thickness": 0.24, # on the far face of the south wall + "profile": quad((11.9, 19.76), (11.0, 19.76), 0.0, 2.1)}, + {"opening_type": "WINDOW", "source_hole_index": 1, "thickness": 0.12, # on the east wall's centre line + "profile": quad((14.06, 21.0), (14.06, 22.2), 0.9, 2.3)}, + {"opening_type": "DOOR", "source_hole_index": 2, "thickness": 0.12, # the next room's wall, 1 m away + "profile": quad((11.0, 19.0), (11.9, 19.0), 0.0, 2.1)}, + {"opening_type": "DOOR", "source_hole_index": 3, "thickness": 0.12, # perpendicular to the west wall + "profile": quad((9.5, 21.0), (9.9, 21.0), 0.0, 2.1)}, + {"opening_type": "HOLE", "source_hole_index": 4, "thickness": 0.24, # passage: not emitted + "profile": quad((11.0, 23.0), (12.0, 23.0), 0.0, 2.1)}] + rep = [{**ops[0], "source_hole_index": 5}, # exact repeat of hole 0 + {**ops[0], "source_hole_index": 6, "profile": quad((11.0, 19.82), (11.9, 19.82), 0.0, 2.1)}, # other face + {**ops[1], "source_hole_index": 7, "profile": quad((14.06, 21.0), (14.06, 22.2), 2.4, 2.7)}] # window above + op, n_rep = openings(ops + rep, ring, "s") + assert n_rep == 2 and [f["furniture_instance_id"] for f in op] == ["s::hole_0", "s::hole_1", "s::hole_7"], op + ir = convert_room({"room_id": "r", "room_boundary": ring, "furniture": op[:2]}, source=SOURCE, + uid="t", group="g", boundary_type="polygon", extra=_extra) + d, w = ir["objects"] + assert d["id"] == "s::hole_0" and d["category"] == "door" and d["structure"] and not d["tilted"], d + assert close(d["size"], [0.9, OPENING_T, 2.1]) and close(d["pos"], [1.45, 0, 0]) and abs(d["yaw"]) < 1e-9, d + assert w["category"] == "window" and w["structure"], w + assert close(w["size"], [1.2, OPENING_T, 1.4]) and close(w["pos"], [4, 1.6, 0.9]) and abs(w["yaw"] - math.pi / 2) < 1e-9, w + # the source's own 'window' box on the same wall line (0.13 m deep) replaces the opening box; a lamp does not + own = {"id": "s::ins_1", "category": "window", "size": [1.2, 0.13, 1.4], "pos": [4, 1.6, 0.9], "yaw": math.pi / 2} + lamp = {**own, "id": "s::ins_2", "category": "lamp"} + assert drop_labelled([d, w, own, lamp]) == [d, own, lamp] and drop_labelled([d, w, lamp]) == [d, w, lamp] + pane = {**own, "size": [0.02, 1.5, 1.6], "pos": [3.97, 1.5, 0.8], "yaw": 0.0} # thin pane 3 cm off the line + sill = {**own, "size": [1.2, 0.5, 0.4], "pos": [3.8, 1.6, 0.0]} # platform below the window + assert drop_labelled([w, pane]) == [pane] and drop_labelled([w, sill]) == [w, sill] + + # wall ring overshooting a corner (crosses itself around a 2.4 x 6.5 cm lobe): repaired, area kept; a bow-tie is not + spur = [[2.912, 0], [2.888, 0], [2.888, 1.678], [0, 1.678], [0, 0.065], [2.912, 0.065]] + fixed, m = repair_ring(spur) + assert not Polygon(spur).is_valid and Polygon(fixed).is_valid and m["boundary_repaired"] == "buffer(0)", m + assert abs(Polygon(fixed).area - abs(signed_area(spur))) < REPAIR_TOL * abs(signed_area(spur)) + bow = [[0, 0], [1, 1], [1, 0], [0, 1]] + assert repair_ring(bow) == (bow, {}) and repair_ring(ring) == (ring, {}) + print("interiorgs.py self-check ok") diff --git a/code/fastfill/adapters/internscenes.py b/code/fastfill/adapters/internscenes.py new file mode 100644 index 0000000000000000000000000000000000000000..4b74619c23ad7ab165304d3db5409905ec9cf386 --- /dev/null +++ b/code/fastfill/adapters/internscenes.py @@ -0,0 +1,183 @@ +"""InternScenes layouts. This module = InternScenes-Gen (SOURCE InternScenes_gen) plus the shared reader for the +Real2Sim re-layouts of real scans (modules internscenes_{scannet,3rscan,arkit,mp3d}). + +Export (kits/extract_internscenes*_json.py): meters, Z-up, bbox = [x, y, z, sx, sy, sz, a_z, a_x, a_y] with +position = box CENTER, sizes along the box local axes, ZXY Euler (R = Rz(a_z) Rx(a_x) Ry(a_y), EmbodiedScan / +pytorch3d convention). Floor is at z = 0. +Boundary: Real2Sim = 2D convex hull of the floor mesh (-> "hull"); Gen = boundary_points.json floor polygon. +Some boxes are stored with a 90/180 deg X/Y rotation (Y-up assets, books lying flat); they are upright boxes +with permuted axes and are re-expressed as yaw-only boxes (`upright`). Boxes tilted by more than TILT_TOL_DEG +are replaced by the yaw-aligned bounding box of their 8 corners (`tilted_bbox`: true footprint, height and bottom, +yaw = heading of the local X axis) with rotation x/y = None so convert_room flags them tilted; "tilt_deg" keeps +the angle between the local axis closest to vertical and world Z. +""" +import csv +import io +import json +import math +import os +import zipfile + +import numpy as np +from shapely.geometry import Polygon + +from fastfill.adapters.unified import convert_room, iter_records + +SOURCE = "InternScenes_gen" +DIR = "imChuling__3D_Room_Collections/InternScenes_exported" +TILT_TOL_DEG = 5.0 +MAX_SIZE_M = 20.0 +# ponytail: one offset for all Real2Sim/Gen assets, set from the inspect_ir front test (see module docstrings) +FRONT_OFFSET_DEG = 0.0 + + +def rot_zxy(az, ax, ay): + cz, sz, cx, sx, cy, sy = math.cos(az), math.sin(az), math.cos(ax), math.sin(ax), math.cos(ay), math.sin(ay) + rz = np.array([[cz, -sz, 0], [sz, cz, 0], [0, 0, 1]]) + rx = np.array([[1, 0, 0], [0, cx, -sx], [0, sx, cx]]) + ry = np.array([[cy, 0, sy], [0, 1, 0], [-sy, 0, cy]]) + return rz @ rx @ ry + + +def upright(R, size, tol_deg=TILT_TOL_DEG): + """World-from-local rotation + local sizes -> (sizes [sx, sy, sz], yaw_deg, k) of the same box as an upright + box, k = index of the source local axis that is vertical; None if no local axis is within tol of vertical. + The new local X is the source local X when it stays horizontal (keeps its front), else the source local Y.""" + R = np.asarray(R, dtype=float) + k = int(np.argmax(np.abs(R[2]))) + if abs(R[2, k]) < math.cos(math.radians(tol_deg)): + return None + hx, hy = [i for i in range(3) if i != k] # k=2: (0,1) k=1: (0,2) k=0: (1,2) + v = R[:, hx] + return [size[hx], size[hy], size[k]], math.degrees(math.atan2(v[1], v[0])), k + + +def tilted_bbox(R, size): + """World-from-local rotation + local sizes of a box that is NOT upright within tolerance -> (sizes [w, l, h] of + the yaw-aligned bounding box of its 8 corners, yaw_deg, k, tilt_deg); k = local axis closest to vertical, tilt = + its angle from world Z; yaw = heading of local X, or of local Y when X is the near-vertical axis (as upright() + does: X's small horizontal part is then only the lean direction, and a box aligned to it is up to 3x too big).""" + R = np.asarray(R, dtype=float) + k = int(np.argmax(np.abs(R[2]))) + a = next((c for c in ((R[:2, 1], R[:2, 0]) if k == 0 else (R[:2, 0], R[:2, 1])) if np.hypot(*c) >= 1e-6), (1.0, 0.0)) + yaw = math.atan2(a[1], a[0]) + D = np.array([[math.cos(yaw), math.sin(yaw), 0], [-math.sin(yaw), math.cos(yaw), 0], [0, 0, 1]]) # u, v, z + ext = np.abs(D @ R) @ np.asarray(size, dtype=float) # max - min of the (origin-symmetric) corners along u, v, z + return ext.tolist(), math.degrees(yaw), k, math.degrees(math.acos(min(1.0, abs(R[2, k])))) + + +def canon(f): + """Exported furniture -> same dict with an upright {x:0,y:0,z:yaw} rotation and permuted sizes when the box is + upright within tolerance; otherwise its yaw-aligned bounding box (`tilted_bbox`) with {x:None,y:None,z:yaw} + (convert_room flags it tilted) and "tilt_deg". The centre is unchanged either way.""" + b = f["source_fields"]["bbox"] + if max(b[3:6]) > MAX_SIZE_M: # corrupt box (2 Gen beds are 1910 m / 803014 m tall) + return {**f, "furniture_size": None} + R = rot_zxy(b[6], b[7], b[8]) + up = upright(R, b[3:6]) + if up is None: + (sx, sy, sz), yaw, k, tilt = tilted_bbox(R, b[3:6]) + rot, extra = {"x": None, "y": None, "z": yaw}, {"tilt_deg": tilt} + else: + (sx, sy, sz), yaw, k = up + rot, extra = {"x": 0.0, "y": 0.0, "z": yaw}, {} + return {**f, "furniture_size": {"width": sx, "length": sy, "height": sz}, "furniture_rotation": rot, + "vertical_axis": k, **extra} + + +def convert(scene, source, uid, group, boundary_type, meta): + room = scene["rooms"][0] + room = {**room, "furniture": [canon(f) for f in room["furniture"]]} + ir = convert_room(room, source=source, uid=uid, group=group, subset=scene.get("source_subset"), + boundary_type=boundary_type, center_z=True, front_offset_deg=FRONT_OFFSET_DEG, + meta={"scene_id": scene["scene_id"], "source_layout": scene["source_layout"], + "source_dataset": scene["source_dataset"], "is_true_room": room.get("is_true_room"), + "tilt_tol_deg": TILT_TOL_DEG, "front_known": True, + "n_permuted_axes": sum(f.get("vertical_axis", 2) != 2 for f in room["furniture"]), + **meta}) + return ir + + +def load_real(root, fname, source, group_of, extra_meta=None): + for s in iter_records(os.path.join(root, DIR, fname)): + yield convert(s, source, uid=f"{source}::{s['scene_id']}", group=group_of(s), boundary_type="hull", + meta=(extra_meta(s) if extra_meta else {})) + + +def rscan_reference(root): + """3RScan scan id (reference or rescan) -> reference scan id.""" + m = json.load(open(os.path.join(root, "imChuling__3D_Room_Collections/3RScan_exported/source_metadata/3RScan.json"))) + ref = {} + for env in m: + ref[env["reference"]] = env["reference"] + for sc in env.get("scans", []): + ref[sc["reference"]] = env["reference"] + return ref + + +def arkit_visits(root): + """ARKitScenes video_id -> visit_id (None when 'NA').""" + z = zipfile.ZipFile(os.path.join(root, "imChuling__3D_Room_Collections/ARKitScenes_exported/source.zip")) + rows = csv.DictReader(io.TextIOWrapper(z.open("source/metadata.csv"), encoding="utf-8")) + return {r["video_id"]: (None if r["visit_id"] in ("", "NA") else r["visit_id"]) for r in rows} + + +def load(root): + """Gen regions: the floor is not at z = 0 (0.10-0.15 m, floor slab); every region has >= 1 door and all + doors of a region share one bottom height, so z is shifted by the median door bottom.""" + for s in iter_records(os.path.join(root, DIR, "internscenes_gen_exported.json")): + rt, rid = s["rooms"][0]["room_type"], s["source_scene_name"] + # boundary_points are wall samples every 0.7 m with the first point repeated: keep the corners only + # (median 27 -> 6 vertices; every polygon is valid, max shift 1 cm) + poly = Polygon(s["rooms"][0]["room_boundary"]).simplify(0.01) + s["rooms"][0]["room_boundary"] = [list(c) for c in poly.exterior.coords[:-1]] + ir = convert(s, SOURCE, uid=f"{SOURCE}::{rt}/{rid}", group=f"internscenes:gen/{rid}", # one id = one generated house across room-type folders + boundary_type="polygon", meta={"asset_source": "Infinigen", "boundary_simplify_m": 0.01}) + doors = sorted(o["pos"][2] for o in ir["objects"] if o["category"] == "door") + fz = doors[len(doors) // 2] if doors else None + for o in ir["objects"]: + o["pos"][2] -= fz or 0.0 + ir["meta"].update(floor_z=fz, floor_z_source="median door bottom" if doors else None) + yield ir + + +if __name__ == "__main__": + # book lying flat: local X vertical -> upright box, height = source width + R = rot_zxy(0.3, 0.0, math.pi / 2) + size, yaw, k = upright(R, [0.03, 0.2, 0.3]) + assert k == 0 and size == [0.2, 0.3, 0.03] and abs(yaw - math.degrees(0.3) - 90) < 1e-6, (size, yaw, k) + # Y-up chair stored with a_x = -90: local Y vertical, local X keeps the yaw + size, yaw, k = upright(rot_zxy(0.0, -math.pi / 2, math.radians(40)), [0.7, 0.8, 0.6]) + assert k == 1 and size == [0.7, 0.6, 0.8] and abs(yaw + 40) < 1e-6, (size, yaw) + assert upright(rot_zxy(0.0, math.radians(10), 0.0), [1, 1, 1]) is None + + # canon: sizes [1.0, 0.2, 0.1], ZXY angles in degrees, centre z + S = [1.0, 0.2, 0.1] + + def rec(ax, ay, az=0.0, z=0.0): + return {"furniture_category": "box", "furniture_position": {"x": 1.0, "y": 1.0, "z": z}, + "source_fields": {"bbox": [1.0, 1.0, z, *S, math.radians(az), math.radians(ax), math.radians(ay)]}} + + def sz(c): + return [c["furniture_size"][key] for key in ("width", "length", "height")] + + # (1) upright box yawed 30 deg: unchanged sizes, yaw 30 + c = canon(rec(0, 0, az=30)) + assert sz(c) == S and abs(c["furniture_rotation"]["z"] - 30) < 1e-9 and "tilt_deg" not in c, c + # (2) tilted 30 deg about X: yaw-aligned bounding box, yaw 0 + s30, c30 = math.sin(math.radians(30)), math.cos(math.radians(30)) + h, l = 0.2 * s30 + 0.1 * c30, 0.2 * c30 + 0.1 * s30 # 0.1866, 0.2232 + c = canon(rec(30, 0)) + assert np.allclose(sz(c), [1.0, l, h]) and abs(c["furniture_rotation"]["z"]) < 1e-9, c + assert c["furniture_rotation"]["x"] is None and c["vertical_axis"] == 2 and abs(c["tilt_deg"] - 30) < 1e-9, c + # (3) lying flat (a_x = 90): permuted upright box via upright(), not the tilted path + c = canon(rec(90, 0)) + assert np.allclose(sz(c), [1.0, 0.1, 0.2]) and c["vertical_axis"] == 1 and "tilt_deg" not in c, c + assert c["furniture_rotation"]["x"] == 0.0 and abs(c["furniture_rotation"]["z"]) < 1e-9, c + # (4) tilted box centred at z = 0.2 -> convert_room bottom = 0.2 - h/2, flagged tilted + scene = {"scene_id": "s", "source_layout": "l", "source_dataset": "d", + "rooms": [{"room_type": "t", "room_boundary": [[0, 0], [4, 0], [4, 4], [0, 4]], "furniture": [rec(30, 0, z=0.2)]}]} + ir = convert(scene, "t", uid="u", group="g", boundary_type="polygon", meta={}) + o = ir["objects"][0] + assert o["tilted"] and abs(o["pos"][2] - (0.2 - h / 2)) < 1e-9 and ir["meta"]["n_tilted"] == 1, (o, ir["meta"]) + print("internscenes.py self-check ok") diff --git a/code/fastfill/adapters/internscenes_3rscan.py b/code/fastfill/adapters/internscenes_3rscan.py new file mode 100644 index 0000000000000000000000000000000000000000..0e668497055af2c1da4adc4899fc1b1d83e8ebe2 --- /dev/null +++ b/code/fastfill/adapters/internscenes_3rscan.py @@ -0,0 +1,12 @@ +"""InternScenes Real2Sim re-layout of 3RScan scans (conventions: see internscenes.py). Group = 3RScan reference scan.""" +from fastfill.adapters.internscenes import load_real, rscan_reference + +SOURCE = "InternScenes_3rscan" + + +def load(root): + ref = rscan_reference(root) + yield from load_real(root, "3rscan_exported.json", SOURCE, + lambda s: f"3rscan:{ref.get(s['source_scene_name'], s['source_scene_name'])}", + lambda s: {"scan_id": s["source_scene_name"], + "is_reference_scan": ref.get(s["source_scene_name"]) == s["source_scene_name"]}) diff --git a/code/fastfill/adapters/internscenes_arkit.py b/code/fastfill/adapters/internscenes_arkit.py new file mode 100644 index 0000000000000000000000000000000000000000..404532545c0d5bef0f59ee727f107262d9146f08 --- /dev/null +++ b/code/fastfill/adapters/internscenes_arkit.py @@ -0,0 +1,14 @@ +"""InternScenes Real2Sim re-layout of ARKitScenes videos (conventions: see internscenes.py). +Group = arkit:; several videos of one visit (same venue) are listed in meta.group_aliases.""" +from fastfill.adapters.internscenes import arkit_visits, load_real + +SOURCE = "InternScenes_arkit" + + +def load(root): + visit = arkit_visits(root) + yield from load_real(root, "arkitscenes_exported.json", SOURCE, + lambda s: "arkit:" + s["source_scene_name"], + lambda s: {"video_id": s["source_scene_name"], "visit_id": visit.get(s["source_scene_name"]), + "group_aliases": [f"arkit_visit:{visit[s['source_scene_name']]}"] + if visit.get(s["source_scene_name"]) else []}) diff --git a/code/fastfill/adapters/internscenes_mp3d.py b/code/fastfill/adapters/internscenes_mp3d.py new file mode 100644 index 0000000000000000000000000000000000000000..d18fa139962da5b49f77bb16d0363ac4218ce010 --- /dev/null +++ b/code/fastfill/adapters/internscenes_mp3d.py @@ -0,0 +1,24 @@ +"""InternScenes Real2Sim re-layout of Matterport3D regions (conventions: see internscenes.py). +One record = one Matterport region (a room-level unit). Group = mp3d:. +InternScenes ships 96 regions twice within a house (e.g. 1LXtFkjw3qL region2 == region29: identical boundary, +same object ids/categories, boxes within 1 cm); only the first copy is emitted (1696 -> 1600 records).""" +import numpy as np + +from fastfill.adapters.internscenes import load_real + +SOURCE = "InternScenes_mp3d" + + +def load(root): + seen = {} + for ir in load_real(root, "matterport3d_exported.json", SOURCE, + lambda s: "mp3d:" + s["source_layout"].split("/")[2], + lambda s: {"region": s["source_scene_name"]}): + key = (ir["group"], tuple((o["id"].rsplit(":", 1)[1], o["category"]) for o in ir["objects"])) + arr = np.array([o["size"] + o["pos"] for o in ir["objects"]], dtype=float) + bnd = np.array(ir["boundary"], dtype=float) + if any(b.shape == bnd.shape and a.shape == arr.shape and np.allclose(b, bnd, atol=0.01) + and np.allclose(a, arr, atol=0.01) for a, b in seen.get(key, [])): + continue + seen.setdefault(key, []).append((arr, bnd)) + yield ir diff --git a/code/fastfill/adapters/internscenes_scannet.py b/code/fastfill/adapters/internscenes_scannet.py new file mode 100644 index 0000000000000000000000000000000000000000..261c24678474347209b90c800ff09e630b04fff5 --- /dev/null +++ b/code/fastfill/adapters/internscenes_scannet.py @@ -0,0 +1,10 @@ +"""InternScenes Real2Sim re-layout of ScanNet v2 scans (conventions: see internscenes.py). +room_type = ScanNet sceneType. Group = scannet:sceneXXXX (rescans _NN share it).""" +from fastfill.adapters.internscenes import load_real + +SOURCE = "InternScenes_scannet" + + +def load(root): + yield from load_real(root, "scannet_exported.json", SOURCE, + lambda s: "scannet:" + s["source_scene_name"].rsplit("_", 1)[0]) diff --git a/code/fastfill/adapters/m3dlayout_infinigen.py b/code/fastfill/adapters/m3dlayout_infinigen.py new file mode 100644 index 0000000000000000000000000000000000000000..669f0a6759de766ab60e82024f29e79ecb45c6cb --- /dev/null +++ b/code/fastfill/adapters/m3dlayout_infinigen.py @@ -0,0 +1,54 @@ +"""M3DLayout, Infinigen subset (procedural houses, one room per record). Aux only: boundary is a proxy. + +Export (extract_m3dlayout_json.py): source Y-up (location = OBB center, size = half extents, rotation = yaw about +Y) +-> furniture_position = (x, -z, y - sy) = BOTTOM center, size = (2sx, 2sz, 2sy), rotation.z = degrees(yaw); +the layout format is yaw-only (detail objects: 1 of 19,643 tilted > 5.7 deg in a 300-scene sample). +XY origin = furniture AABB SW corner; room_boundary = that AABB (object_extent_aabb) -> boundary_type "proxy"; +room_height is an object-extent proxy too -> None. The layout frame is the Infinigen scene mirrored in Y +(layout_y = C - native_y, checked against the detail objects); a mirrored room is still a valid layout. +Front = layout local -Y (front test vs the proxy rectangle: toilet +Y 0.84 / -Y 0.00, bed 0.73 / 0.00). +room_type = M3DLayout's explicit-keyword parse of the source description (None / 'unknown' -> None). +Records without layout furniture (detail-only) are skipped; repeated (scene_id, layout) records are emitted once. +""" +import os + +from fastfill.adapters.unified import convert_room, iter_records + +SOURCE = "M3DLayout_infinigen" +DIR = "imChuling__3D_Room_Collections/M3DLayout_exported" + + +def furniture(f): + """Layout furniture -> unified schema: yaw-only format means upright; snake_case category -> words.""" + return {**f, "furniture_category": f["furniture_category"].replace("_", " "), + "furniture_rotation": {**f["furniture_rotation"], "x": 0, "y": 0}} + + +def records(root, name): + """(scene, room) for records that have layout furniture, each distinct (scene_id, layout) once.""" + seen = set() + for sc in iter_records(os.path.join(root, DIR, name)): + room = sc["rooms"][0] + if not room["furniture"]: + continue + key = (sc["scene_id"], tuple(sorted((f["furniture_category"], round(f["furniture_position"]["x"], 3), + round(f["furniture_position"]["y"], 3), round(f["furniture_position"]["z"], 3), + round(f["furniture_rotation"]["z"], 1)) for f in room["furniture"]))) + if key in seen: + continue + seen.add(key) + yield sc, room + + +def room_type(room): + return None if room.get("room_type") in (None, "unknown") else room["room_type"].replace("_", " ") + + +def load(root): + for sc, room in records(root, "infinigen.json"): + yield convert_room({**room, "room_height": None, "room_type": room_type(room), + "furniture": [furniture(f) for f in room["furniture"]]}, + source=SOURCE, uid=sc["sample_id"], group="infinigen:" + sc["scene_id"].split("_", 1)[1], + subset=sc["source_subset"], boundary_type="proxy", front_offset_deg=-90, + meta={"scene_id": sc["scene_id"], "front_known": True, "mirrored_frame": True, + "room_type_source": room.get("room_type_source"), "height_source": "dropped_proxy"}) diff --git a/code/fastfill/adapters/mansionworld.py b/code/fastfill/adapters/mansionworld.py new file mode 100644 index 0000000000000000000000000000000000000000..0a1cee4ed37305b873a5ce0c9f9d80a29bc5f8f4 --- /dev/null +++ b/code/fastfill/adapters/mansionworld.py @@ -0,0 +1,234 @@ +"""MansionWorld (AI2-THOR / Objathor procedural buildings), room view json_room.json (27,884 rooms, 1,000 building +variants of 300 floor plans; building/floor views are the same rooms regrouped). + +Export (kits/convert_mansionworld.py, enrich_wall_objects.py): meters, Z-up, (x, z)_yup -> (x, -z), room polygon +recentred on its area centroid. Floor objects: position = THOR position (asset pivot; footprint center within 5 cm), +z = 0 (bottom), size = padded cm footprint minus 0.10 m, de-rotated (yaws are multiples of 90), height = 2 * center y. +Yaw = THOR rotation.y, which the (x, -z) map turns into CCW yaw about +Z of asset local +X; front = Unity +Z = local -Y; +a full THOR rotation (x, y, z) is rot_zxy(y, x, -z) in this frame (Unity applies z, x, y; checked to 0.0). +Exception: the single custom asset of `toilet_suite` (a stall) is authored backwards: its +Y side touches the wall +and its paired toilet_suite_door lies on the room side, so it gets +180 deg. +Wall objects: z = bottom; xy were recentred on the vertex mean instead of the area centroid (bug), corrected here; +width/length had the floor-footprint padding subtracted although wall footprints are unpadded (bug), added back here. +Surface (small) objects: source_fields.surface_groups (one per parent floor object, parent id = support_surfaces +parent_object_id) -> anchor "object", parent = that id, IR id "|" (a third of the small names repeat +under another parent of the same room; the 22 exported objects whose raw id repeats get "#k"). Room xy = parent +position + R(parent yaw) position_local and yaw = parent yaw + yaw_deg_local, both equal to the raw position / +rotation.y (3,355 objects, max diff 0.0). +The export keeps only rotation.y; the raw floor_*.json small_objects (same download, joined by "|" +within the room) give x/z: 92.2 % of 201,302 are 0, the rest exact 90-deg axis flips (pens, notebooks, phones, frames +lying flat), none at another angle. Flipped boxes become upright boxes with permuted sizes (internscenes.upright; +counted in meta.n_permuted_axes); their source front is vertical, so their yaw is only a box orientation. The export +bottom (support height_m + z_local = raw center y - exported height / 2) is corrected by (exported - true height) / 2: +flipped bottoms then match the nearest upright neighbour on the same parent to 0.2 mm median (was 6.6 cm). Surface +objects without a raw record are not emitted (meta.surface_objects_excluded). build.prep keeps the ones whose bottom +is within 5 cm of the parent box top (the rest stand on shelves / inside furniture). +Desc: the export's Objathor asset description of floor, wall and surface objects is passed as "desc". +Room type: every raw floor has exactly one roomType "corridor" room (3,719 of 3,719 floors), its open hub, whatever +it holds (living_dining_room, open_office_area, master_bedroom, hub_room, ...); for these the room name (room_id minus +"F_" = raw portable_source_id, 27,884 of 27,884) is the room type. The source value is kept in meta.room_type_src. +Doors / windows: room["doors"] / ["windows"] = {center_xy, width_m, ...} from the raw doorSegment / windowSegment, +already in the room's recentred frame (all door centres lie on the polygon ring). Emitted as "structure" boxes: +local X along the wall = opening width, 0.10 m across the wall, yaw = CCW direction of the nearest polygon edge. +Windows: z = sill_height_m, height = height_m (from the raw holePolygon). Doors: the export dropped the hole height; +raw holePolygon / door-database.json give 2.07-2.13 m for every door asset, so DOOR_H = 2.1 (clearance_depth_m is +the swing clearance, not a thickness). Openings > 0.15 m off the ring (1.8 % of windows: same-named room on another +floor, matched by the converter with the F_ prefix stripped) are dropped and counted in meta.n_openings_dropped. +""" +import collections +import functools +import json +import math +import os +import re + +from shapely.geometry import LineString, Point + +from fastfill.adapters.internscenes import rot_zxy, upright +from fastfill.adapters.unified import _num, convert_room, iter_records, signed_area + +SOURCE = "MansionWorld" +DIR = "imChuling__3D_Room_Collections/MansionWorld_exported" +FLOOR_PREFIX = re.compile(r"^F\d+_") +PAD, MIN_DIM = 0.10, 0.01 # enrich_wall_objects.py FOOTPRINT_PADDING_M / MIN_DIM_M +DOOR_H, OPENING_T, OFF_WALL = 2.1, 0.10, 0.15 + + +def fix(f, vx, vy): + f = dict(f) + if f["source_fields"].get("mount_type") == "wall" and f.get("furniture_position"): + p = f["furniture_position"] + f["furniture_position"] = {**p, "x": p["x"] + vx, "y": p["y"] + vy} + # enrich_wall_objects.py also subtracts the 0.10 m floor-footprint padding, but wall footprints are unpadded + # (raw sample: footprint == annotation bbox for 1,652/1,652). Values clamped to 0.01 stay (true extent <= 0.11). + s = f.get("furniture_size") + if s: + f["furniture_size"] = {**s, **{k: s[k] + PAD for k in ("width", "length") if s[k] > MIN_DIM + 1e-6}} + if f["furniture_category"] == "toilet_suite" and f.get("furniture_rotation"): + f["furniture_rotation"] = {**f["furniture_rotation"], "z": f["furniture_rotation"]["z"] + 180} + if f["furniture_category"].startswith("vertical_core_"): # stair / elevator cores: let the structure filter see them + f["furniture_category"] = f["furniture_category"].replace("_", " ") + return f + + +def openings(room): + """Doors / windows -> (furniture dicts on the wall line, n_dropped). Rotation z = wall direction + 90 so that + convert_room(front_offset_deg=-90) yields yaw = wall direction and size = [width_m, OPENING_T, height].""" + b = room.get("room_boundary") or [] + if len(b) < 3: + return [], 0 + if signed_area(b) < 0: + b = b[::-1] + edges = [LineString(s) for s in zip(b, b[1:] + b[:1])] + out, dropped = [], 0 + for kind, key, items in (("door", "door_id", room.get("doors")), ("window", "window_id", room.get("windows"))): + for e in items or []: + x, y = e.get("center_xy") or (None, None) + z, h = (0.0, DOOR_H) if kind == "door" else (e.get("sill_height_m"), e.get("height_m")) + edge = min(edges, key=lambda ln: ln.distance(Point(x, y))) if _num(x) and _num(y) else None + if not all(_num(v) for v in (x, y, e.get("width_m"), z, h)) or edge.distance(Point(x, y)) > OFF_WALL: + dropped += 1 + continue + (x0, y0), (x1, y1) = edge.coords + out.append({"furniture_category": kind, "furniture_instance_id": e[key], "structure": True, + "furniture_position": {"x": x, "y": y, "z": z}, + "furniture_rotation": {"x": 0.0, "y": 0.0, "z": math.degrees(math.atan2(y1 - y0, x1 - x0)) + 90}, + "furniture_size": {"width": OPENING_T, "length": e["width_m"], "height": h}}) + return out, dropped + + +@functools.lru_cache(maxsize=2) # json_room.json is grouped by building +def raw_rotations(root, building): + """{room id: {"|": [raw rotation {x, y, z}, ...]}} from the building's floor_*.json small_objects + (id "| ()"; roomId may lack the F_ prefix, which is unique per floor once stripped).""" + out, d = {}, os.path.join(root, DIR, "data/MansionWorld/mansionworld", building) + for fn in os.listdir(d): + if fn.startswith("floor_") and fn.endswith(".json"): + with open(os.path.join(d, fn)) as fh: + raw = json.load(fh) + rid = {FLOOR_PREFIX.sub("", r["id"]): r["id"] for r in raw["rooms"]} + for s in raw.get("small_objects") or []: + r = rid.get(FLOOR_PREFIX.sub("", s.get("roomId", ""))) + if r: + out.setdefault(r, {}).setdefault(s["id"].split(" (", 1)[0], []).append(s["rotation"]) + return out + + +def surface(room, rots): + """surface_groups -> (furniture dicts with "parent", n without a raw rotation); rots = raw_rotations()[room id]. + Positions / yaws are composed from the parent's unmodified export pose (see module doc).""" + furn = {f["furniture_instance_id"]: f for f in room["furniture"]} + tops = {t["surface_id"]: t for t in room["source_fields"].get("support_surfaces") or []} + out, missing, seen = [], 0, collections.Counter() + for g in room["source_fields"].get("surface_groups") or []: + top = tops[g["surface_id"]] + pid = top["parent_object_id"] + pp, pyaw = furn[pid]["furniture_position"], furn[pid]["furniture_rotation"]["z"] + c, s = math.cos(math.radians(pyaw)), math.sin(math.radians(pyaw)) + for o in g["objects"]: + key = f"{o['object_id']}|{pid}" + seen[key] += 1 + rs = rots.get(key, []) + if len(rs) < seen[key]: + missing += 1 + continue + r = rs[seen[key] - 1] + R = rot_zxy(*(math.radians(v) for v in (pyaw + o["yaw_deg_local"], r["x"], -r["z"]))) + (sx, sy, sz), yaw, k = upright(R, o["dimensions"]) + (lx, ly), h = o["position_local"], o["dimensions"][2] + out.append({"furniture_category": o["category"], "parent": pid, "description": o.get("description"), + "furniture_instance_id": key if seen[key] == 1 else f"{key}#{seen[key]}", + "furniture_position": {"x": pp["x"] + c * lx - s * ly, "y": pp["y"] + s * lx + c * ly, + "z": top["height_m"] + o["z_local"] + (h - sz) / 2}, + "furniture_rotation": {"x": 0.0, "y": 0.0, "z": yaw}, + "furniture_size": {"width": sx, "length": sy, "height": sz}, "vertical_axis": k}) + return out, missing + + +def extra(f): + if f.get("structure"): + return {"structure": True} + if f.get("parent"): + return {"anchor": "object", "parent": f["parent"], "desc": f.get("description")} + wall = f["source_fields"].get("mount_type") == "wall" + return {"anchor": "wall" if wall else "floor", "desc": f.get("description")} + + +def to_ir(room, scene, rots): + prov = scene["provenance"] + building = prov["source_house_id"] + b = room.get("room_boundary") or [] + vx, vy = (sum(p[0] for p in b) / len(b), sum(p[1] for p in b) / len(b)) if b else (0.0, 0.0) + boxes, n_dropped = openings(room) + smalls, missing = surface(room, rots) + rt = room.get("room_type") + return convert_room( + {**room, "furniture": [fix(f, vx, vy) for f in room["furniture"]] + smalls + boxes, + "room_type": FLOOR_PREFIX.sub("", room["room_id"]) if rt == "corridor" else rt}, source=SOURCE, + uid=f"MansionWorld::{scene['scene_id']}", group="mansionworld:" + building.split("#")[0], + boundary_type="polygon", center_z=False, front_offset_deg=-90, extra=extra, + meta={"building": building, "room_id": room["room_id"], "room_type_src": rt, + "geometry_hash": prov.get("geometry_hash"), "front_known": True, "n_openings_dropped": n_dropped, + "surface_objects_excluded": missing, "n_permuted_axes": sum(f["vertical_axis"] != 2 for f in smalls), + "license_note": "CC BY 4.0 + HF gate: non-commercial research"}) + + +def load(root): + for scene in iter_records(os.path.join(root, DIR, "json_room.json")): + rots = raw_rotations(root, scene["provenance"]["source_house_id"]) + for room in scene["rooms"]: + yield to_ir(room, scene, rots.get(room["room_id"], {})) + + +if __name__ == "__main__": + from fastfill.anchors import fixed_geometry + scene = {"scene_id": "s", "provenance": {"source_house_id": "b#0"}} + room = {"room_id": "r", "room_type": "office", "room_height": 3.2, "furniture": [], + "room_boundary": [[-1, -1], [1, -1], [1, 1], [-1, 1]], "source_fields": {}, + "doors": [{"door_id": "d", "center_xy": [1.0, 0.0], "width_m": 0.9, "clearance_depth_m": 0.9}], + "windows": [{"window_id": "w", "center_xy": [0.0, 1.0], "width_m": 1.2, "sill_height_m": 0.9, "height_m": 1.2}, + {"window_id": "far", "center_xy": [-5.0, 1.0], "width_m": 1.2, "sill_height_m": 0.9, "height_m": 1.2}]} + ir = to_ir(room, scene, {}) + d, w = ir["objects"] + assert d["category"] == "door" and d["structure"] and d["size"] == [0.9, OPENING_T, DOOR_H], d + assert d["pos"] == [2.0, 1.0, 0.0] and abs(d["yaw"] - math.pi / 2) < 1e-9, d # right wall runs +Y (CCW) + assert w["size"] == [1.2, OPENING_T, 1.2] and w["pos"] == [1.0, 2.0, 0.9] and abs(w["yaw"] - math.pi) < 1e-9, w + assert ir["meta"]["n_openings_dropped"] == 1 and ir["meta"]["n_incomplete"] == 0 + assert [o["id"] for o in fixed_geometry(ir)] == ["d", "w"] + assert ir["room_type"] == "office" and ir["meta"]["room_type_src"] == "office" + + # surface objects: desk (THOR yaw 90) at (0.5, 0.2) holding an upright cup, a notebook stored Z-up and flipped + # x = 90 (exported dims [w, d, h] = [0.2, 0.02, 0.3]: its 0.02 side is the vertical one) and two "pen-0" whose + # raw id repeats; a pen on a shelf has no raw record. Export bottom = height_m + z_local = raw y - dims[2] / 2. + desk = {"furniture_category": "desk", "furniture_instance_id": "desk-0", "description": "A wooden desk.", + "furniture_position": {"x": 0.5, "y": 0.2, "z": 0.0}, "furniture_rotation": {"x": 0, "y": 0, "z": 90}, + "furniture_size": {"width": 1.2, "length": 0.6, "height": 0.75}, "source_fields": {"mount_type": "floor"}} + obj = lambda n, dims, loc, yl, zl: {"object_id": n, "category": n[:-2], "dimensions": dims, "position_local": loc, + "yaw_deg_local": yl, "z_local": zl, "description": f"A {n[:-2]}."} + room = {**room, "room_id": "F2_living_dining_room", "room_type": "corridor", "doors": [], "windows": [], + "furniture": [desk], "source_fields": { + "support_surfaces": [{"surface_id": "desk-0/top", "parent_object_id": "desk-0", "height_m": 0.75}], + "surface_groups": [{"surface_id": "desk-0/top", "objects": [ + obj("cup-0", [0.1, 0.08, 0.12], [0.2, 0.0], 10.0, 0.0), + obj("notebook-0", [0.2, 0.02, 0.3], [-0.3, 0.1], 0.0, -0.14), + obj("pen-0", [0.14, 0.01, 0.01], [0.0, 0.0], 0.0, 0.0), + obj("pen-0", [0.14, 0.01, 0.01], [0.1, 0.0], 0.0, 0.0), + obj("pen-1", [0.14, 0.01, 0.01], [0.0, 0.0], 0.0, -0.4)]}]}} + rots = {"cup-0|desk-0": [{"x": 0, "y": 100, "z": 0}], "notebook-0|desk-0": [{"x": 90, "y": 90, "z": 0}], + "pen-0|desk-0": [{"x": 0, "y": 90, "z": 0}] * 2} + ir = to_ir(room, scene, rots) + assert ir["room_type"] == "living_dining_room" and ir["meta"]["room_type_src"] == "corridor", ir["room_type"] + assert ir["meta"]["surface_objects_excluded"] == 1 and ir["meta"]["n_permuted_axes"] == 1, ir["meta"] + by = {o["id"]: o for o in ir["objects"]} + assert list(by) == ["desk-0", "cup-0|desk-0", "notebook-0|desk-0", "pen-0|desk-0", "pen-0|desk-0#2"], list(by) + assert by["desk-0"]["anchor"] == "floor" and by["desk-0"]["desc"] == "A wooden desk." + close = lambda u, v: all(abs(x - y) < 1e-9 for x, y in zip(u, v)) + cup, nb = by["cup-0|desk-0"], by["notebook-0|desk-0"] + assert (cup["anchor"], cup["parent"], cup["desc"], cup["tilted"]) == ("object", "desk-0", "A cup.", False), cup + # R(90) (0.2, 0) = (0, 0.2): room (0.5, 0.4) -> AABB-shifted by (-1, -1); yaw 90 + 10 - 90 (front offset); d, w, h + assert close(cup["pos"], [1.5, 1.4, 0.75]) and abs(cup["yaw"] - math.radians(10)) < 1e-9, cup + assert close(cup["size"], [0.08, 0.1, 0.12]), cup + # R(90) (-0.3, 0.1) = (-0.1, -0.3); lying flat: true height 0.02, bottom 0.61 + (0.3 - 0.02) / 2 = 0.75 + assert close(nb["pos"], [1.4, 0.9, 0.75]) and close(nb["size"], [0.3, 0.2, 0.02]), nb + assert abs(math.sin(nb["yaw"])) < 1e-9 < math.cos(nb["yaw"]) and nb["parent"] == "desk-0", nb # heading of local X + print("mansionworld.py self-check ok") diff --git a/code/fastfill/adapters/multiscan.py b/code/fastfill/adapters/multiscan.py new file mode 100644 index 0000000000000000000000000000000000000000..be5a570d4d14af0e1cb1f510bd71a9889db5829d --- /dev/null +++ b/code/fastfill/adapters/multiscan.py @@ -0,0 +1,120 @@ +"""MultiScan (XXXpilar/multiscan-clean repackaging): 273 scans of 117 real spaces, one scan = one space. + +objects.csv: obb_center (box CENTER), obb_half_extents (HALF sizes along the box axes), obb_axes = 9 floats = +the three box axes one after another (world directions), plus annotated `front` / `up` vectors; `up` is box +axis 1 for 10,957/10,957 objects and `front` is one of the box axes. Z-up meters, scan-local frame whose floor +is NOT at z = 0. regions.csv floor_height is the TOP of the floor-slab AABB (slabs are 0.07-0.46 m thick scan +noise) and furniture bottoms sit a median 0.09 m below it, so the floor level used is the median bottom of the +objects that relations.csv marks rests_on_floor (>= 1 per scan; geometric relation of the repackaging). +Boundary: regions.csv poly_loop is the convex hull of the floor-slab OBBs, but floors are annotated only where +the scan saw them (22% of in-scope objects stick out of it by a median 0.24 m). The boundary used is the convex +hull of poly_loop + the wall-segment endpoints (structure only) -> "hull" when every floor-standing +non-architectural object fits inside it (+5 cm); otherwise the room gets boundary_type "proxy" with that hull +extended by those objects' footprints (object extent, cannot prove in-bounds validity). +height = ceiling_height - floor level, only when height_reliable. Group = multiscan: (rescans _NN share it). +""" +import csv +import json +import math +import os + +import numpy as np +from shapely.geometry import MultiPoint, Polygon + +from fastfill.adapters.internscenes import TILT_TOL_DEG, tilted_bbox, upright +from fastfill.adapters.unified import convert_room +from fastfill.scene import footprint + +SOURCE = "MultiScan" +DIR = "XXXpilar__multiscan-clean" + + +def box(o): + """objects.csv row -> unified furniture dict (bottom-center position, upright yaw from the annotated front).""" + axes = np.array(json.loads(o["obb_axes"]), dtype=float).reshape(3, 3) # rows = box axes + size = [2 * v for v in json.loads(o["obb_half_extents"])] + c = json.loads(o["obb_center"]) + fr = np.array(json.loads(o["front"]), dtype=float) + d = axes @ fr + f = int(np.argmax(np.abs(d))) + order = [f] + [i for i in range(3) if i != f] # front axis first = local +X + R = np.stack([axes[i] * (np.sign(d[f]) if i == f else 1) for i in order], axis=1) + up = upright(R, [size[i] for i in order]) + if up is None: # tilted: keep a non-zero tilt + sz, yaw, k, tilt = tilted_bbox(R, [size[i] for i in order]) + rot = {"x": tilt, "y": 0.0, "z": yaw} + else: + sz, yaw, k = up + rot = {"x": 0.0, "y": 0.0, "z": yaw} + bottom = c[2] - sz[2] / 2 + return {"furniture_category": o["category"].replace("_", " "), "furniture_instance_id": f"o{o['object_id']}", + "furniture_position": {"x": c[0], "y": c[1], "z": bottom}, "furniture_rotation": rot, + "furniture_size": {"width": sz[0], "length": sz[1], "height": sz[2]}, + "front_vertical": k == 0, + "structure": o["is_architectural"] == "True" or o["is_opening"] == "True"} + + +def boundary(g, furn, tol=0.05): + """(polygon, type, n_outside) - see module docstring.""" + walls = json.loads(g["wall_segments"] or "[]") + pts = json.loads(g["poly_loop"]) + [w[k] for w in walls for k in ("start", "end")] + hull = MultiPoint([tuple(p) for p in pts]).convex_hull + fps = [Polygon(footprint({"pos": [f["furniture_position"]["x"], f["furniture_position"]["y"]], + "size": [f["furniture_size"]["width"], f["furniture_size"]["length"]], + "yaw": math.radians(f["furniture_rotation"]["z"])})) + for f in furn if not f["structure"] and f["furniture_position"]["z"] < tol] + out = [fp for fp in fps if not hull.buffer(tol).contains(fp)] + if out: + hull = MultiPoint(list(hull.exterior.coords) + [c for fp in out for c in fp.exterior.coords]).convex_hull + return [list(c) for c in hull.exterior.coords[:-1]], ("proxy" if out else "hull"), len(out) + + +def load(root): + base = os.path.join(root, DIR) + scans = {r["scan_id"]: r for r in csv.DictReader(open(os.path.join(base, "scans.csv")))} + objs, on_floor = {}, {} + for o in csv.DictReader(open(os.path.join(base, "objects.csv"))): + objs.setdefault(o["scan_id"], []).append(o) + for r in csv.DictReader(open(os.path.join(base, "relations.csv"))): + if r["relation"] == "rests_on_floor": + on_floor.setdefault(r["scan_id"], set()).add(r["subject_id"]) + for g in csv.DictReader(open(os.path.join(base, "regions.csv"))): + sid, s = g["scan_id"], scans[g["scan_id"]] + bottoms = [json.loads(o["aabb_min"])[2] for o in objs.get(sid, []) if o["object_id"] in on_floor.get(sid, ())] + fz = float(np.median(bottoms)) if bottoms else float(g["floor_height"]) + furn = [box(o) for o in objs.get(sid, [])] + for f in furn: + f["furniture_position"]["z"] -= fz + poly, btype, n_out = boundary(g, furn) + # source flags is_architectural (wall/floor/ceiling/beam/pillar/windowsill/...) and is_opening are kept as + # "structure": build never places them but must see them (a floor pillar rejects the room) + room = {"room_id": sid, "room_type": s["room_type"].lower(), "room_boundary": poly, + # extrusion_height = ceiling - floor_height (slab top); measure from the floor level used for z + "room_height": float(g["ceiling_height"]) - fz if g["height_reliable"] == "True" else None, + "furniture": furn} + yield convert_room(room, source=SOURCE, uid=f"{SOURCE}::{sid}", group=f"multiscan:{s['scene_id']}", + boundary_type=btype, extra=lambda f: { + **({"structure": True} if f["structure"] else {}), + **({"front_known": False} if f["front_vertical"] else {})}, + meta={"scan_id": sid, "scene_id": s["scene_id"], "device": s["device"], + "floor_z": fz, "floor_z_source": "median bottom of rests_on_floor objects" if bottoms + else "regions.floor_height", "floor_height": float(g["floor_height"]), "poly_source": g["poly_source"], + "boundary_source": "convex hull of floor slabs + wall segments" + + (" + floor objects outside it" if n_out else ""), "n_floor_objects_outside_structure": n_out, + "height_reliable": g["height_reliable"] == "True", "front_known": True, + "front_source": "annotated front vector", "tilt_tol_deg": TILT_TOL_DEG, + "n_front_vertical": sum(f["front_vertical"] for f in furn)}) + + +if __name__ == "__main__": + # bag from scene_00000_00: up = axis 1, front = axis 2 -> yaw of the front vector, size [2*h2, 2*h0, 2*h1] + row = {"obb_axes": "[0.9861942396120026, 0.16559263798884283, 0, 0, 0, 1, 0.16559263798884283, -0.9861942396120026, 0]", + "obb_half_extents": "[0.126, 0.166, 0.0795]", "obb_center": "[0, 0, 1]", "category": "shoe_box", + "front": "[0.16559263798884283, -0.9861942396120026, 0]", "object_id": "2", + "is_architectural": "False", "is_opening": "False"} + b = box(row) + assert b["furniture_category"] == "shoe box" and b["furniture_rotation"]["x"] == 0 + assert abs(b["furniture_rotation"]["z"] - math.degrees(math.atan2(-0.98619, 0.16559))) < 1e-3 + assert [round(v, 3) for v in b["furniture_size"].values()] == [0.159, 0.252, 0.332] + assert abs(b["furniture_position"]["z"] - (1 - 0.166)) < 1e-9 + print("multiscan.py self-check ok") diff --git a/code/fastfill/adapters/optiscene_holodeck.py b/code/fastfill/adapters/optiscene_holodeck.py new file mode 100644 index 0000000000000000000000000000000000000000..da0402c707acaf484739b4dd98d288eb1f6dee4d --- /dev/null +++ b/code/fastfill/adapters/optiscene_holodeck.py @@ -0,0 +1,45 @@ +"""OptiScene / 3D-SynthPlace, HOLODECK subset: 9,977 generated single rooms (Objaverse assets, floor objects only). + +Raw fields (source_fields.layout_object, see OptiScene/README.md and gprompt.py): Unity/AI2-THOR frame = y-up, +LEFT-handed; position {x, y, z} m with y = 0 for every object (bottom) and x/z = box center; rotation.y = yaw in +degrees (clockwise seen from above); bbox = [height, width, depth] cm along the mesh y, x, z. +LH (x, y, z) -> RH Z-up (x, z, y) is a proper change of frame (what the export stored), but the yaw sign flips: +yaw of mesh +X in the IR = -rotation.y. Holodeck meshes face mesh -Z = IR local -Y -> front_offset_deg = -90. +The export's furniture_rotation.z (= +rotation.y) and size mapping are therefore not used. +""" +import os + +from fastfill.adapters.unified import convert_room, iter_records + +SOURCE = "OptiScene_holodeck" +OPTI = "imChuling__3D_Room_Collections/OptiScene_exported" + + +def raw_furniture(f): + lo = f["source_fields"]["layout_object"] + p, r, b = lo.get("position") or {}, lo.get("rotation") or {}, lo.get("bbox") + return {**f, + "furniture_position": {"x": p.get("x"), "y": p.get("z"), "z": p.get("y")}, + "furniture_rotation": {"x": r.get("x", 0), "y": r.get("z", 0), + "z": -r["y"] if isinstance(r.get("y"), (int, float)) else None}, + "furniture_size": {"width": b[1] / 100, "length": b[2] / 100, "height": b[0] / 100} if b and len(b) == 3 else None} + + +def load(root): + for scene in iter_records(os.path.join(root, OPTI, "holodeck.json")): + for room in scene["rooms"]: + yield convert_room({**room, "furniture": [raw_furniture(f) for f in room["furniture"]]}, + source=SOURCE, uid=f"optiscene:{scene['scene_id']}", + group=f"holodeck:{scene['scene_id']}", subset="HOLODECK", boundary_type="polygon", + front_offset_deg=-90, extra=lambda f: {"desc": f.get("description")}, + meta={"source_file": scene["provenance"]["source_file"], "front_known": True, + "size_note": "Holodeck bbox, padded ~11 cm in width/depth (see report)"}) + + +if __name__ == "__main__": + f = {"source_fields": {"layout_object": {"position": {"x": 1, "y": 0, "z": 2}, "rotation": {"x": 0, "y": 90, "z": 0}, + "bbox": [75, 77, 174]}}} + g = raw_furniture(f) + assert g["furniture_position"] == {"x": 1, "y": 2, "z": 0} and g["furniture_rotation"]["z"] == -90 + assert g["furniture_size"] == {"width": 0.77, "length": 1.74, "height": 0.75} + print("optiscene_holodeck self-check ok") diff --git a/code/fastfill/adapters/sage10k.py b/code/fastfill/adapters/sage10k.py new file mode 100644 index 0000000000000000000000000000000000000000..9f7a1e133e132f175c81e90961c185cc67799c18 --- /dev/null +++ b/code/fastfill/adapters/sage10k.py @@ -0,0 +1,255 @@ +"""SAGE-10k (agentic generation, TRELLIS assets): 9,987 single-room scenes + 13 multi-room scenes (40 rooms). + +Aggregate json_single.json / json_multi_split.json (aggregate_scenes.py): meters, Z-up, room_boundary = ordered wall +centreline endpoints (all rooms are 4-corner rectangles), position = bottom center (floor objects z = 0, equals the +cm footprint center of placement_constraints), rotation = Euler deg applied as Rz Ry Rx (kits/tex_utils_local.py), +dimensions = local X/Y/Z extents. Front = local +Y (front test), so front_offset_deg = 90. +Support: the aggregate drops place_id; it is read from the per-scene layout_*.json when present +(floor | wall | ), otherwise anchors are left to fastfill.anchors. +Doors: the aggregate drops walls, so doors come from the per-scene layout (same frame): centre = wall start + +position_on_wall * (end - start), bottom = wall base z, extent width x wall thickness x height (kits/tex_utils_local.py +create_door_mesh). Emitted as structure boxes. Layout `windows` are empty in the export; windows exist only as +furniture objects, which are kept as they are. +Descriptions: every layout object has one ("description"); passed through as desc when the layout is id-aligned. +Tilt: floor/wall objects are exactly upright; objects resting on objects carry x/y tilt from the physics check. +Tilt <= TILT_DEG is treated as settling noise (zeroed); an axis permutation within TILT_DEG (1,237 objects turned +90 deg onto a side) is an exact upright box. A larger tilt becomes the yaw-aligned upright bounding box of the 8 +corners (`upright_box`), flagged tilted, with tilt_deg and the source rotation rot_src kept. The rotation is about the +object origin = bottom centre of its local box (SAGE server code, EmbodiedGen/gentask_baseline/sage/server: +objects/object_generation.py centres x/y and sets min z = 0; the physics check returns that origin's transform, +isaac_sim_mcp_extension/extension.py, scipy 'xyz' = Rz Ry Rx). +Support (`rest`): yet the stored pose keeps the origin on the support whatever the tilt, so the rotated box sinks in: +over 11,855 tilted (>= 5 deg) children whose parent top is confirmed by an untilted sibling, origin - top has median +-0.3..+0.4 cm in every lift bin up to 12 cm (corr 0.07), and a wine bottle lying on a hutch has its axis on the hutch +top. The box is kept whole (the source's extents) and stood on its support: its bottom is raised to the place_id +parent's top, or to the origin (the stored contact height) when the parent top is higher (a shelf or recess inside the +parent's box); "z_lift" records the rise. Build's 5 cm top-surface test then sees the source's contact height, not a +corner that sank 4-9 cm into the desk. Boxes turned exactly onto a side rest on their lowest face (90% of 107 on +confirmed tops), so min(parent top, origin) leaves them as they are. +""" +import collections +import json +import math +import os + +import numpy as np + +from fastfill.adapters.internscenes import tilted_bbox, upright +from fastfill.adapters.unified import convert_room, iter_records + +SOURCE = "SAGE-10k" +TILT_DEG = 2.0 + + +def tilt_deg(rot): + return math.degrees(math.acos(max(-1.0, min(1.0, math.cos(math.radians(rot["x"])) * math.cos(math.radians(rot["y"])))))) + + +def rot_zyx(r): + """Euler deg {x, y, z} -> world-from-local R = Rz Ry Rx.""" + (cx, sx), (cy, sy), (cz, sz) = [(math.cos(math.radians(r[a])), math.sin(math.radians(r[a]))) for a in "xyz"] + return np.array([[cz, -sz, 0], [sz, cz, 0], [0, 0, 1]]) @ np.array([[cy, 0, sy], [0, 1, 0], [-sy, 0, cy]]) @ \ + np.array([[1, 0, 0], [0, cx, -sx], [0, sx, cx]]) + + +def upright_box(f, cut=None): + """Tilted furniture (Euler deg Rz Ry Rx about the bottom centre of its local box, size = local X/Y/Z extents, + front = local +Y; SAGE and SceneSmith) -> the same dict as an upright box, front passed as local X to the + internscenes rules (yaw = front heading, or local X heading when the front is the near-vertical axis): + a local axis within TILT_DEG of vertical -> the exact box with permuted sizes (`upright`), rotation {0, 0, yaw - 90}, + not tilted; otherwise the yaw-aligned bounding box of the 8 corners (as `tilted_bbox`) minus the part below `cut` + (the height below which the source says the object has nothing: its support), rotation {x: None, y: None, + z: yaw - 90} so that convert_room(front_offset_deg=90) gives yaw back and flags it tilted. Position = bottom centre + of the new box; "_tilt" = provenance (rot_src, and tilt_deg for tilted boxes).""" + r, p, s = f.get("furniture_rotation"), f.get("furniture_position"), f.get("furniture_size") + if not (r and p and s): + return f + R, S, o = rot_zyx(r)[:, [1, 0, 2]], [s["length"], s["width"], s["height"]], np.array([p["x"], p["y"], p["z"]]) + up = upright(R, S, TILT_DEG) + if up: + (l, w, h), yaw, _ = up + c = o + R[:, 2] * S[2] / 2 # box centre + return {**f, "furniture_position": {"x": float(c[0]), "y": float(c[1]), "z": float(c[2]) - h / 2}, + "furniture_rotation": {"x": 0, "y": 0, "z": yaw - 90}, + "furniture_size": {"width": w, "length": l, "height": h}, "_tilt": {"rot_src": dict(r)}} + _, yaw, _, tilt = tilted_bbox(R, S) + C = np.array([[x, y, z] for x in (-S[0] / 2, S[0] / 2) for y in (-S[1] / 2, S[1] / 2) for z in (0, S[2])]) @ R.T + z0 = cut - o[2] if cut is not None and C[:, 2].min() < cut - o[2] < C[:, 2].max() else C[:, 2].min() + edges = [(i, i | b) for i in range(8) for b in (1, 2, 4) if not i & b] + P = [c for c in C if c[2] >= z0] + [C[i] + (C[j] - C[i]) * (z0 - C[i, 2]) / (C[j, 2] - C[i, 2]) + for i, j in edges if (C[i, 2] - z0) * (C[j, 2] - z0) < 0] + D = np.array([[math.cos(math.radians(yaw)), math.sin(math.radians(yaw)), 0], + [-math.sin(math.radians(yaw)), math.cos(math.radians(yaw)), 0], [0, 0, 1]]) # u, v, z rows + lo, hi = (np.array(P) @ D.T).min(0), (np.array(P) @ D.T).max(0) + b = o + D.T @ [(lo[0] + hi[0]) / 2, (lo[1] + hi[1]) / 2, lo[2]] # bottom centre + return {**f, "furniture_position": {"x": float(b[0]), "y": float(b[1]), "z": float(b[2])}, + "furniture_rotation": {"x": None, "y": None, "z": yaw - 90}, + "furniture_size": {"width": float(hi[1] - lo[1]), "length": float(hi[0] - lo[0]), + "height": float(hi[2] - lo[2])}, + "_tilt": {"tilt_deg": tilt, "rot_src": dict(r)}} + + +def rest(f, parent): + """upright_box(f) stood on its support (module doc): its bottom raised to min(place_id parent's top, origin z), never + lowered. The box as it is when the parent is missing or not upright (its box top is no surface).""" + g = upright_box(f) + if parent is None or tilt_deg(parent["furniture_rotation"]) > TILT_DEG: + return g + s = min(parent["furniture_position"]["z"] + parent["furniture_size"]["height"], f["furniture_position"]["z"]) + b = g["furniture_position"]["z"] + if b >= s: + return g + return {**g, "furniture_position": {**g["furniture_position"], "z": s}, "_tilt": {**g["_tilt"], "z_lift": s - b}} + + +def doors(lroom): + """Layout doors as unified-schema furniture. convert_room(front_offset_deg=90) swaps width/length and adds 90 deg, + so width=thickness / length=door width / rotation=wall-90 come out as size [door width, thickness, height] with + yaw = wall direction (long side along the wall).""" + walls = {w["id"]: w for w in lroom.get("walls") or []} + out = [] + for d in lroom.get("doors") or []: + w = walls.get(d.get("wall_id")) + if w is None: + continue + s, e, t = w["start_point"], w["end_point"], d["position_on_wall"] + out.append({"furniture_category": "door", "furniture_instance_id": d["id"], "_ir": {"structure": True}, + "furniture_position": {"x": s["x"] + (e["x"] - s["x"]) * t, "y": s["y"] + (e["y"] - s["y"]) * t, + "z": s["z"]}, + "furniture_rotation": {"x": 0, "y": 0, + "z": math.degrees(math.atan2(e["y"] - s["y"], e["x"] - s["x"])) - 90}, + "furniture_size": {"width": w.get("thickness") or 0.10, "length": d["width"], "height": d["height"]}}) + return out + + +def load(root): + base = os.path.join(root, "liantian__3D_Room_Scene_Collections/SAGE10k") + sdir = os.path.join(base, "sage10k/scenes") + layouts = {d.split("_", 2)[-1]: os.path.join(sdir, d, d.split("_", 2)[-1] + ".json") + for d in (os.listdir(sdir) if os.path.isdir(sdir) else [])} + for fn in ("json_single.json", "json_multi_split.json"): + for rec in iter_records(os.path.join(base, fn)): + sid, k = rec["scene_id"], rec["room_index"] + fids = [f["furniture_instance_id"] for f in rec["furniture"]] + count = collections.Counter(fids) + place, lroom = None, {} + if sid in layouts and os.path.exists(layouts[sid]): + with open(layouts[sid]) as fh: + lroom = json.load(fh)["rooms"][k] + if [o["id"] for o in lroom["objects"]] == fids: + place = [o.get("place_id") for o in lroom["objects"]] + n_noise = 0 + furn, seen = [], collections.Counter() + for j, f in enumerate(rec["furniture"]): + p = place[j] if place else None + if p in ("floor", "wall"): + sup = {"anchor": p} + elif count.get(p) == 1: + sup = {"anchor": "object", "parent": p} + else: + sup = {} # no evidence, dangling or repeated parent id: inferred downstream + if place is not None and lroom["objects"][j].get("description"): + sup["desc"] = lroom["objects"][j]["description"] + f = {**f, "_ir": sup} + seen[fids[j]] += 1 + if seen[fids[j]] > 1: # raw ids repeat for copied small objects (663 rooms): keep ids unique + f["furniture_instance_id"] = f"{fids[j]}#{seen[fids[j]]}" + r = f.get("furniture_rotation") + if r and (r["x"] or r["y"]): + if tilt_deg(r) <= TILT_DEG: + f["furniture_rotation"] = {"x": 0, "y": 0, "z": r["z"]} + n_noise += 1 + else: + f = rest(f, rec["furniture"][fids.index(p)] if sup.get("parent") else None) + furn.append(f) + furn += doors(lroom) + + yield convert_room( + {**rec, "furniture": furn, "room_boundary": [[q["x"], q["y"]] for q in rec.get("room_boundary") or []]}, + source=SOURCE, uid=f"SAGE-10k::{sid}::{k}", group=f"sage:{sid}", + subset="multi" if fn == "json_multi_split.json" else "single", boundary_type="polygon", + center_z=False, front_offset_deg=90, extra=lambda f: {**f["_ir"], **f.get("_tilt", {})}, + meta={"scene_id": sid, "room_index": k, "front_known": True, "tilt_threshold_deg": TILT_DEG, + "n_tilt_noise_zeroed": n_noise, "anchor_source": "place_id" if place is not None else "inferred", + "building_style": rec.get("building_style")}) + + +if __name__ == "__main__": + # 4 x 3 room at (10, 20); door on the bottom wall (+X direction), door on the right wall (+Y direction) + walls = [{"id": f"w{i}", "start_point": {"x": a[0], "y": a[1], "z": 0.0}, "end_point": {"x": b[0], "y": b[1], "z": 0.0}, + "thickness": 0.1} for i, (a, b) in enumerate([((10, 20), (14, 20)), ((14, 20), (14, 23))])] + lroom = {"walls": walls, "doors": [{"id": "d0", "wall_id": "w0", "position_on_wall": 0.5, "width": 0.9, "height": 2.1}, + {"id": "d1", "wall_id": "w1", "position_on_wall": 0.25, "width": 0.8, "height": 2.0}, + {"id": "dx", "wall_id": "missing", "position_on_wall": 0.5, "width": 1, "height": 2}]} + ir = convert_room({"room_boundary": [[10, 20], [14, 20], [14, 23], [10, 23]], "furniture": doors(lroom)}, + source=SOURCE, uid="t", group="g", boundary_type="polygon", center_z=False, front_offset_deg=90, + extra=lambda f: f["_ir"]) + a, b = ir["objects"] + close = lambda u, v: all(abs(x - y) < 1e-6 for x, y in zip(u, v)) + assert a["category"] == "door" and a["structure"] and not a["tilted"], a + assert close(a["pos"], [2, 0, 0]) and close(a["size"], [0.9, 0.1, 2.1]) and abs(math.sin(a["yaw"])) < 1e-9, a + assert close(b["pos"], [4, 0.75, 0]) and close(b["size"], [0.8, 0.1, 2.0]) and abs(b["yaw"] - math.pi / 2) < 1e-9, b + + # tilted box -> upright bounding box. Size [1.0, 0.2, 0.1] (front = local +Y), origin (1, 1, 0.5), 30 deg about X: + # IR size [along front, width, height] = [0.2 c + 0.1 s, 1.0, 0.2 s + 0.1 c], yaw 90, bottom centre (1, 0.975, 0.45) + def ir_obj(f, cut=None): + return convert_room({"room_boundary": [[0, 0], [9, 0], [9, 9], [0, 9]], "furniture": [upright_box(f, cut)]}, + source=SOURCE, uid="t", group="g", boundary_type="polygon", center_z=False, + front_offset_deg=90, extra=lambda f: {**f["_ir"], **f.get("_tilt", {})})["objects"][0] + + def rec(r, S, p=(1, 1, 0.5)): + return {"furniture_category": "book", "furniture_position": dict(zip("xyz", p)), "furniture_rotation": r, + "furniture_size": dict(zip(("width", "length", "height"), S)), "_ir": {"desc": "a book"}} + s30, c30 = 0.5, math.cos(math.radians(30)) + x30 = rec({"x": 30, "y": 0, "z": 0}, [1.0, 0.2, 0.1]) + o = ir_obj(x30) + assert o["tilted"] and close(o["size"], [0.2 * c30 + 0.1 * s30, 1.0, 0.2 * s30 + 0.1 * c30]), o + assert abs(o["yaw"] - math.pi / 2) < 1e-9 and close(o["pos"], [1, 0.975, 0.45]) and o["desc"] == "a book", o + assert abs(o["tilt_deg"] - 30) < 1e-9 and o["rot_src"] == {"x": 30, "y": 0, "z": 0}, o + # resting on a support at the origin (0.5) or 2 cm below it: the corners under it are cut, footprint unchanged + for cut in (0.5, 0.48): + o = ir_obj(x30, cut) + assert close(o["size"], [0.2 * c30 + 0.1 * s30, 1.0, 0.1 * s30 + 0.1 * c30 + 0.5 - cut]), o + assert close(o["pos"], [1, 0.975, cut]) and o["tilted"], o + assert close(ir_obj(x30, 0.3)["pos"], [1, 0.975, 0.45]) and close(ir_obj(x30, 0.9)["pos"], [1, 0.975, 0.45]) # not through it + # standing on the support: raised to a parent top at/below the origin, to the origin under a higher top (also when + # rolled onto its side), never lowered, not raised under a tilted or missing parent + par = lambda t, r=None: {"furniture_position": {"x": 1, "y": 1, "z": t - 0.4}, "furniture_size": {"height": 0.4}, + "furniture_rotation": r or {"x": 0, "y": 0, "z": 10}} + rolled, side = rec({"x": 70, "y": 0, "z": 0}, [1.0, 0.2, 0.1]), rec({"x": 90, "y": 0, "z": 0}, [1.0, 0.2, 0.1]) + for f, pa, z in ((x30, par(0.48), 0.48), (x30, par(0.53), 0.5), (x30, par(0.3), 0.45), (x30, None, 0.45), + (x30, par(0.48, {"x": 20, "y": 0, "z": 0}), 0.45), (rolled, par(0.53), 0.5), + (rolled, par(0.48), 0.48), (side, par(0.4), 0.4)): # exact side box on its lowest face: as is + g, h = rest(f, pa), upright_box(f) + assert g["furniture_size"] == h["furniture_size"] and g["furniture_position"]["x"] == h["furniture_position"]["x"], g + assert abs(g["furniture_position"]["z"] - z) < 1e-9, (f, pa, g) + assert abs(g["_tilt"].get("z_lift", 0) - (z - h["furniture_position"]["z"])) < 1e-9, g + # 90 deg about X (front vertical): exact upright box, sizes [1.0, 0.1, 0.2], yaw 0, centre (1, 0.95, 0.5), not tilted + o = ir_obj(rec({"x": 90, "y": 0, "z": 0}, [1.0, 0.2, 0.1])) + assert not o["tilted"] and "tilt_deg" not in o and o["rot_src"]["x"] == 90, o + assert close(o["size"], [1.0, 0.1, 0.2]) and close(o["pos"], [1, 0.95, 0.4]) and abs(math.sin(o["yaw"])) < 1e-9, o + # any rotation and cut: the IR box contains every point of the source box above the cut, is tight to it (points + # sampled on the 12 edges, where the clipped box's vertices lie), its bottom is the cut, and its +X is the front's + # heading unless the front (local Y) is the near-vertical axis + rng = np.random.default_rng(0) + edges = [(i, i | b) for i in range(8) for b in (1, 2, 4) if not i & b] + t = np.linspace(0, 1, 401)[:, None] + for k in range(400): + r, S = dict(zip("xyz", rng.uniform(-180, 180, 3))), rng.uniform(0.02, 1.0, 3) + R = rot_zyx(r) + W = np.array([[x, y, z] for x in (-S[0] / 2, S[0] / 2) for y in (-S[1] / 2, S[1] / 2) for z in (0, S[2])]) @ R.T + cut = 0.7 + rng.uniform(W[:, 2].min(), W[:, 2].max()) if k % 2 else None + o = ir_obj(rec(r, S, (4, 4, 0.7)), cut) + if not o["tilted"]: + continue # an axis within TILT_DEG of vertical: exact box, checked above + E = np.vstack([W[i] + t * (W[j] - W[i]) for i, j in edges]) + [4, 4, 0.7] + E = E[E[:, 2] >= (cut if cut is not None else -1.0)] + cy, sy = math.cos(o["yaw"]), math.sin(o["yaw"]) + L = (E - o["pos"]) @ np.array([[cy, sy, 0], [-sy, cy, 0], [0, 0, 1]]).T + lo, hi = np.array([-o["size"][0] / 2, -o["size"][1] / 2, 0]), np.array([o["size"][0] / 2, o["size"][1] / 2, o["size"][2]]) + assert (L.min(0) > lo - 1e-9).all() and (L.max(0) < hi + 1e-9).all(), (r, S, cut, o) + assert np.allclose(L.min(0), lo, atol=3e-3) and np.allclose(L.max(0), hi, atol=3e-3), (r, S, cut, o) + assert cut is None or abs(o["pos"][2] - cut) < 1e-9, (r, S, cut, o) + if np.argmax(np.abs(R[2])) != 1: + assert abs((math.atan2(R[1, 1], R[0, 1]) - o["yaw"] + math.pi) % (2 * math.pi) - math.pi) < 1e-9, (r, o) + print("sage10k.py self-check ok") diff --git a/code/fastfill/adapters/scan2cad.py b/code/fastfill/adapters/scan2cad.py new file mode 100644 index 0000000000000000000000000000000000000000..bfed6b4ff547f249fb674d473d95b5380b96dd47 --- /dev/null +++ b/code/fastfill/adapters/scan2cad.py @@ -0,0 +1,96 @@ +"""Scan2CAD CAD alignments on ScanNet v2 scans (SceneCAD_Scan2CAD_exported). Auxiliary tier. + +Export (kits aggregate_scenecad_scan2cad.py): meters, Z-up, ScanNet axisAlignment-applied scan frame. +scan2cad_alignments[].furniture_position = oriented CAD bbox CENTER (position_type), size = full CAD bbox +extents along the object's canonical Z-up axes [2*bx*sx, 2*bz*sz, 2*by*sy], rotation = Rz*Ry*Rx Euler of +rotation_matrix_zup_scan_local, whose columns are the object's local X / Y / Z (local +Y = ShapeNet -Z). +Front: ShapeNet canonical front = CAD -Z = local +Y (front test: at -90 beds/bookshelves touched the wall with +X), so front_offset_deg = +90. + +Boundary: the SceneCAD floor plane polygon (human layout annotation, 994 scans) -> "polygon"; otherwise the +exported convex hull of ScanNet floor vertices, which misses occluded floor (38% of CAD footprints leave it vs 14% +for the SceneCAD polygon, mostly < 10 cm wall penetration) and so is typed "proxy". +Limitations (why B_aux): one record = one whole ScanNet scan (is_true_room=false), and only the ~30% of objects +that were matched to a ShapeNet CAD are present. +Scan data noise: per-object tilt median 3 deg (9-DoF fit); tilts <= MAX_TILT_DEG are treated as upright. +Bottoms scatter +-10 cm around the floor, so z is measured from the per-scan median bottom of floor-standing +categories and bottoms within FLOOR_SNAP of it are set to 0. Only bottoms within FLOOR_BAND of z = 0 vote: the +axisAlignment frame already puts the floor there (SceneCAD floor plane z p1..p99 = -0.20..0.07), and an unbanded +median followed stacked/wall-mounted CADs (scene0074_01: one shelf at 2.0 m -> floor_z 2.0; 47 scans |floor_z| > 0.2). CAD fits with an axis < 1 mm (a trs.scale axis of +1e-14..1e-8, 75 objects, mostly displays) have no usable size: their size is dropped, so the room counts as incomplete. +""" +import math +import os +import statistics + +from shapely.geometry import Polygon + +from fastfill.adapters.unified import convert_room, iter_records + +SOURCE = "Scan2CAD" +PATH = "imChuling__3D_Room_Collections/SceneCAD_Scan2CAD_exported/scenecad_scan2cad_combined.json" +MAX_TILT_DEG = 10.0 # p95 of |tilt| over all 14,225 alignments is 9.7 deg +FLOOR_SNAP = 0.15 +FLOOR_BAND = 0.25 +FLOOR_CATS = {"chair", "table", "sofa", "bed", "bookshelf", "file_cabinet", "trash_can", "washer", + "dishwasher", "stove", "piano", "bench"} + + +def tilt_deg(a): + return math.degrees(math.acos(max(-1.0, min(1.0, a["source_fields"]["rotation_matrix_zup_scan_local"][2][2])))) + + +def shell_floor(shell): + """XY ring of the single horizontal SceneCAD plane at the bottom of the shell, or None.""" + if not shell: + return None + v = shell["vertices_zup_m"] + zmin = min(p[2] for p in v) + lows = [] + for q in shell["planes"]: + a, b, c = (v[i] for i in q[:3]) + n = [(b[1] - a[1]) * (c[2] - a[2]) - (b[2] - a[2]) * (c[1] - a[1]), + (b[2] - a[2]) * (c[0] - a[0]) - (b[0] - a[0]) * (c[2] - a[2]), + (b[0] - a[0]) * (c[1] - a[1]) - (b[1] - a[1]) * (c[0] - a[0])] + if abs(n[2]) > 0.9 * math.hypot(*n) and sum(v[i][2] for i in q) / len(q) < zmin + 0.3: + lows.append([[v[i][0], v[i][1]] for i in q]) + if len(lows) != 1 or not Polygon(lows[0]).is_valid: + return None + return lows[0] + + +def load(root): + for s in iter_records(os.path.join(root, PATH)): + room = s["rooms"][0] + al = room["scan2cad_alignments"] + if not al: # scenecad_only scans: shell without objects + continue + bottoms = [a["furniture_position"]["z"] - a["furniture_size"]["height"] / 2 for a in al + if a["furniture_category"] in FLOOR_CATS and tilt_deg(a) <= MAX_TILT_DEG] + bottoms = [b for b in bottoms if abs(b) < FLOOR_BAND] + floor = statistics.median(bottoms) if bottoms else None + furn, snapped = [], 0 + for a in al: + p, r = a["furniture_position"], a["furniture_rotation"] + z = p["z"] - a["furniture_size"]["height"] / 2 - (floor or 0.0) + if floor is not None and abs(z) < FLOOR_SNAP: + z, snapped = 0.0, snapped + 1 + upright = tilt_deg(a) <= MAX_TILT_DEG + degenerate = min(a["furniture_size"].values()) < 1e-3 + furn.append({**a, "furniture_size": None if degenerate else a["furniture_size"], "furniture_position": {"x": p["x"], "y": p["y"], "z": z}, + "furniture_rotation": {"x": 0 if upright else r["x"], "y": 0 if upright else r["y"], "z": r["z"]}}) + sid = s["scene_id"] + poly = shell_floor(room["scene_shell"]) + yield convert_room( + {**room, "furniture": furn, "room_boundary": poly or room["room_boundary"]}, source=SOURCE, uid=f"{SOURCE}::{sid}", group=f"scannet:{sid.rsplit('_', 1)[0]}", + subset=s.get("source_subset"), boundary_type="polygon" if poly else "proxy", center_z=False, front_offset_deg=90, + extra=lambda f: {"sym": f["source_fields"]["scan2cad_annotation"]["sym"]}, + meta={"scene_id": sid, "front_known": True, "is_true_room": False, "boundary_source": "scenecad_floor_plane" if poly else room["room_boundary"] and "scannet_floor_vertex_convex_hull", + "source_coverage": s["provenance"]["source_coverage"], "floor_z": floor, "n_floor_snapped": snapped}) + + +if __name__ == "__main__": + shell = {"vertices_zup_m": [[0, 0, 0], [4, 0, 0], [4, 3, 0], [0, 3, 0], [0, 0, 2.5], [4, 0, 2.5]], + "planes": [[0, 1, 2, 3], [0, 1, 5, 4]]} # floor quad + one wall + assert shell_floor(shell) == [[0, 0], [4, 0], [4, 3], [0, 3]] + assert shell_floor({**shell, "planes": [[0, 1, 5, 4]]}) is None + print("scan2cad.py self-check ok") diff --git a/code/fastfill/adapters/scannetpp.py b/code/fastfill/adapters/scannetpp.py new file mode 100644 index 0000000000000000000000000000000000000000..21aad888178a951ff5cc0beeb413ce9af87e6277 --- /dev/null +++ b/code/fastfill/adapters/scannetpp.py @@ -0,0 +1,45 @@ +"""ScanNet++ v2 instance OBBs (ScanNet++_exported). Auxiliary tier: NO semantic front. + +Export (kits/convert_scannetpp.py): meters, Z-up, scene-local; furniture_position = OBB centroid; size = OBB +axesLengths reordered so the axis closest to world Z is local Z (frame kept right-handed); rotation = Rz*Ry*Rx +Euler of that frame. The OBBs are fitted per instance, so local X is an arbitrary horizontal box axis: yaw is +the box orientation modulo the box symmetry, not a facing direction (meta front_known=False on every room). +Boundary = XY AABB of all floor-OBB cross-sections (a superset of the floor; whole apartments stay one record). +Tilts <= MAX_TILT_DEG (vertical_axis_dot_z) are treated as upright; z is measured from the per-scene 5th percentile +bottom of all upright boxes and bottoms within FLOOR_SNAP of it are set to 0. (A median over "floor" categories +followed wall cabinets/shelves: b73f5cdc41 floor 1.46 m; against the annotated thin floor OBBs it was > 0.2 m off in +108/748 scenes, the p5 bottom in 5/772.) +""" +import math +import os + +from fastfill.adapters.unified import convert_room, iter_records + +SOURCE = "ScanNetpp" +PATH = "BillLin66__3D_Room_Collections/ScanNet++_exported/scannetpp.jsonl" +MAX_TILT_DEG = 10.0 +FLOOR_SNAP = 0.10 +COS_TILT = math.cos(math.radians(MAX_TILT_DEG)) + + +def load(root): + for s in iter_records(os.path.join(root, PATH)): + room = s["rooms"][0] + upright = [f["source_fields"]["vertical_axis_dot_z"] >= COS_TILT for f in room["furniture"]] + bottoms = sorted(f["furniture_position"]["z"] - f["furniture_size"]["height"] / 2 + for f, u in zip(room["furniture"], upright) if u) + floor = bottoms[len(bottoms) // 20] if bottoms else None + furn, snapped = [], 0 + for f, u in zip(room["furniture"], upright): + p, r = f["furniture_position"], f["furniture_rotation"] + z = p["z"] - f["furniture_size"]["height"] / 2 - (floor or 0.0) + if floor is not None and abs(z) < FLOOR_SNAP: + z, snapped = 0.0, snapped + 1 + furn.append({**f, "furniture_position": {"x": p["x"], "y": p["y"], "z": z}, + "furniture_rotation": {"x": 0 if u else r["x"], "y": 0 if u else r["y"], "z": r["z"]}}) + sid = s["scene_id"].removeprefix("scannetpp_") + yield convert_room( + {**room, "furniture": furn}, source=SOURCE, uid=f"{SOURCE}::{sid}", group=f"scannetpp:{sid}", + subset=s.get("source_subset"), boundary_type="hull", center_z=False, + meta={"scene_id": sid, "front_known": False, "is_true_room": False, + "boundary_source": room["room_boundary"] and "floor_obb_xy_aabb", "floor_z": floor, "n_floor_snapped": snapped}) diff --git a/code/fastfill/adapters/scenesmith.py b/code/fastfill/adapters/scenesmith.py new file mode 100644 index 0000000000000000000000000000000000000000..d609c31245b881a400a5e8229c234b90dd289a5e --- /dev/null +++ b/code/fastfill/adapters/scenesmith.py @@ -0,0 +1,129 @@ +"""SceneSmith example scenes (179 Room scenes + 31 Houses split into 185 rooms). Eval-only: too small, and +they are the paper's evaluation scenes (prompts 0-99 are the SceneEval-100 benchmark). + +Export (kits/scenesmith_aggregate.py): meters, Z-up, furniture_position = DMD model-frame origin, rotation = +AngleAxis -> XYZ Euler deg (R = Rz Ry Rx), size = asset glTF AABB in the Z-up asset frame (x, y, z). +Asset origin, checked on geometry_metadata AABBs (not in the aggregate): furniture/manipuland = bottom center; +wall_mounted = back face (bbox y in [0, L]), vertical center; ceiling_mounted = top center. +Front = local +Y (wall-mounted assets extend into +Y; front test on beds/wardrobes). +Boundary = floor-plan AABB; every room has exactly north/south/east/west walls, so it is the real rectangle. +Free bodies are physics-settled: x/y tilt <= TILT_DEG is settling noise and is zeroed; an axis permutation within +TILT_DEG is an exact upright box; a larger tilt (only furniture and manipulands, whose origin is the bottom centre) +becomes the yaw-aligned upright bounding box of the 8 corners (sage10k.upright_box: same rotation order, origin and +front), flagged tilted, with tilt_deg and rot_src kept. Unlike SAGE the boxes rest on their lowest corner (1,135 tilted +manipulands over furniture tops: lowest corner - top median -0.1..-1.1 cm in every lift bin), so they are not moved. +Composite members are cut at the bottom of what holds them (scenesmith agent_utils/room.py to_drake_directive): a fill +item `_f_` is inside the container `_f_c` of the same room, a stack member `_s_` sits +on `_s_`, so nothing of it is below that holder's bottom; yet 205 of 865 tilted fill items and 11 of 278 tilted +stack members reach below it (round fruit, markers leaning in cups, a knife on a cutting board). The holder is listed first +(1,199 of 1,199 fill items, 1,079 of 1,079 stack members), so fastfill.anchors, which visits boxes by bottom height, +settles the holder before them instead of standing it on what it holds. +Openings: the export has no doors, and its 910 windows carry no location: every window position is the room frame +origin (= boundary centre, z = 0; the window glTF AABB is centred on its own origin and the placing pose lives in the +room_geometry SDF, which is not exported). Only the window count is kept (meta.n_windows_unlocated), no box is invented. +""" +import math +import os +import re + +from fastfill.adapters.sage10k import upright_box +from fastfill.adapters.unified import convert_room, iter_records + +SOURCE = "SceneSmith" +TILT_DEG = 2.0 +ANCHOR = {"furniture": "floor", "wall_mounted": "wall", "ceiling_mounted": "ceiling"} # manipuland: inferred + + +def tilt_deg(rot): + return math.degrees(math.acos(max(-1.0, min(1.0, math.cos(math.radians(rot["x"])) * math.cos(math.radians(rot["y"])))))) + + +def fix(f, cut=None): + """Origin -> bottom center, settling tilt -> upright, larger tilt -> upright bounding box cut at `cut` (the + bottom of the composite member holding it). Returns (furniture dict, noise_zeroed).""" + rot, p, s, g = f.get("furniture_rotation"), f.get("furniture_position"), f.get("furniture_size"), f["source_fields"]["asset_group"] + if not (rot and p and s): + return f, False + p = dict(p) + yaw = math.radians(rot["z"]) + if g == "wall_mounted": + p["x"] -= math.sin(yaw) * s["length"] / 2 + p["y"] += math.cos(yaw) * s["length"] / 2 + p["z"] -= s["height"] / 2 + elif g == "ceiling_mounted": + p["z"] -= s["height"] + f = {**f, "furniture_position": p} + if not (rot["x"] or rot["y"]): + return f, False + if tilt_deg(rot) <= TILT_DEG: + return {**f, "furniture_rotation": {"x": 0, "y": 0, "z": rot["z"]}}, True + return upright_box(f, cut), False + + +def fix_room(furniture): + """fix() over one room in order, each composite member cut at the bottom of its holder (module doc; the export + lists the holder first): fill item `_f_` -> container `_f_c`, stack member `_s_` -> `_s_`.""" + fixed, done = [], {} + for f in furniture: + m = re.search(r"_([fs]\d+)_(c|\d+)$", f.get("furniture_instance_id") or "") + g, i = m.groups() if m else (None, None) + k = (g, "c") if m and g[0] == "f" and i != "c" else (g, str(int(i) - 1)) if m and g[0] == "s" and i != "0" \ + else None + h = done.get(k) + fixed.append(fix(f, h["furniture_position"]["z"] if h else None)) + if m: + done[g, i] = fixed[-1][0] + return fixed + + +def load(root): + base = os.path.join(root, "imChuling__3D_Room_Collections/SceneSmith_exported") + for fn in ("json_single.json", "multiroom/multiroom_split.json"): + for scene in iter_records(os.path.join(base, fn)): + parent = scene["parent_scene_id"] + for room in scene["rooms"]: + fixed = fix_room(room["furniture"]) + room = {**room, "furniture": [f for f, _ in fixed]} + ir = convert_room( + room, source=SOURCE, uid=f"SceneSmith::{scene['scene_id']}", group=f"scenesmith:{parent}", + subset=scene["provenance"]["subset"], boundary_type="polygon", center_z=False, + front_offset_deg=90, + extra=lambda f: {"anchor": ANCHOR.get(f["source_fields"]["asset_group"]), + "asset_group": f["source_fields"]["asset_group"], **f.get("_tilt", {})}, + meta={"eval_only": True, "front_known": True, "file": fn, "scene_id": scene["scene_id"], + "parent_scene_id": parent, "tilt_threshold_deg": TILT_DEG, + "n_tilt_noise_zeroed": sum(z for _, z in fixed), "room_type_source": "dmd_room_frame", + "n_windows_unlocated": len(room.get("windows") or [])}) + yield ir + + +if __name__ == "__main__": + def rec(g, rot, p=(1.0, 1.0, 0.0), S=(1.0, 0.2, 0.1)): + return {"furniture_category": "x", "source_fields": {"asset_group": g}, "furniture_position": dict(zip("xyz", p)), + "furniture_rotation": rot, "furniture_size": dict(zip(("width", "length", "height"), S))} + # settling noise (1 deg) -> zeroed, sizes kept + f, z = fix(rec("manipuland", {"x": 1.0, "y": 0.0, "z": 30.0})) + assert z and f["furniture_rotation"] == {"x": 0, "y": 0, "z": 30.0} and f["furniture_size"]["length"] == 0.2, f + # 30 deg about X -> upright bounding box [0.2 c + 0.1 s, 1.0, 0.2 s + 0.1 c] (front first), yaw 90, flagged tilted + f, z = fix(rec("manipuland", {"x": 30.0, "y": 0.0, "z": 0.0}, p=(1.0, 1.0, 0.5))) + ir = convert_room({"room_boundary": [[0, 0], [4, 0], [4, 4], [0, 4]], "furniture": [f]}, source=SOURCE, uid="t", + group="g", boundary_type="polygon", front_offset_deg=90, extra=lambda f: f.get("_tilt", {})) + o, c30 = ir["objects"][0], math.cos(math.radians(30)) + assert not z and o["tilted"] and abs(o["tilt_deg"] - 30) < 1e-9 and abs(o["yaw"] - math.pi / 2) < 1e-9, o + assert all(abs(a - b) < 1e-9 for a, b in zip(o["size"] + o["pos"], [0.2 * c30 + 0.05, 1.0, 0.1 + 0.1 * c30, + 1.0, 0.975, 0.45])), o + # wall-mounted: back-face / vertical-centre origin -> bottom centre (yaw 90: +Y local = -X world) + f, _ = fix(rec("wall_mounted", {"x": 0, "y": 0, "z": 90.0}, p=(2.0, 1.0, 1.5), S=(0.6, 0.1, 0.4))) + assert all(abs(f["furniture_position"][k] - v) < 1e-9 for k, v in zip("xyz", (1.95, 1.0, 1.3))), f + # tilted composite members are cut at their holder's bottom (0.48): the fill item of container f2 and stack member + # s1_1 on s1_0; not an item of a group without a holder here (f3), a stack base, a plain manipuland + box = rec("manipuland", {"x": 0, "y": 0, "z": 0}, p=(1.0, 1.0, 0.48)) + item = rec("manipuland", {"x": 30.0, "y": 0, "z": 0}, p=(1.0, 1.0, 0.5)) + ids = ("s::bowl_0_f2_c", "s::apple_0_f2_0", "s::apple_1_f3_0", "s::pear_0", "s::board_0_s1_0", "s::knife_0_s1_1", + "s::book_0_s4_0") + got = [f for f, _ in fix_room([{**g, "furniture_instance_id": i} + for g, i in zip((box, item, item, item, box, item, item), ids)])] + assert all(abs(g["furniture_position"]["z"] - z) < 1e-9 + for g, z in zip(got, (0.48, 0.48, 0.45, 0.45, 0.48, 0.48, 0.45))), got + assert abs(got[1]["furniture_size"]["height"] - (0.05 + 0.1 * c30 + 0.02)) < 1e-9, got[1] + print("scenesmith.py self-check ok") diff --git a/code/fastfill/adapters/spatialgen.py b/code/fastfill/adapters/spatialgen.py new file mode 100644 index 0000000000000000000000000000000000000000..c54b6bd9aa0ec33eb690cdca0dc1fdad515a851b --- /dev/null +++ b/code/fastfill/adapters/spatialgen.py @@ -0,0 +1,19 @@ +"""SpatialGen public test set (48 designer rooms). Eval-only: the source release is a test split. + +Export: Z-up meters, scene-local, furniture_position = box center, size = scaled local XYZ extents, +yaw about +Z (deg). Boundary = floor polygon. Front = local -Y (front test: beds/wardrobes touch the wall with +Y). +""" +import os + +from fastfill.adapters.unified import convert_room, iter_records + +SOURCE = "SpatialGen" + + +def load(root): + path = os.path.join(root, "BillLin66__3D_Room_Collections/SpatialGen_exported/spatialgen.jsonl") + for scene in iter_records(path): + for room in scene["rooms"]: + yield convert_room(room, source=SOURCE, uid=room["room_id"], group=f"spatialgen:{scene['scene_id']}", + subset="test", boundary_type="polygon", center_z=True, + front_offset_deg=-90, meta={"eval_only": True, "front_known": True}) diff --git a/code/fastfill/adapters/spatiallm.py b/code/fastfill/adapters/spatiallm.py new file mode 100644 index 0000000000000000000000000000000000000000..fd4b149520ffadf8b639e404ce2ccbdc6e89063f --- /dev/null +++ b/code/fastfill/adapters/spatiallm.py @@ -0,0 +1,177 @@ +"""SpatialLM-Dataset layouts (Manycore/Kujiale designer houses: 12,328 scenes, 54,775 rooms). +View used: spatiallm.json only (scene records, every room once); json_room.json and multiroom/*.json are +re-packagings of the same rooms (split_views.py) and are ignored. + +Export (aggregate_spatiallm.py): source Bbox(category, px,py,pz, angle_z, sx,sy,sz) -> furniture_position = box +CENTER, furniture_size = full scale along local x/y/z, rotation.z = degrees(angle_z); the format is yaw-only, so +the exported x/y = null mean "not tilted" (set to 0 here). Meters, Z-up, scene-global XY. +room_boundary = ring of wall start points (wall_polygon). 2152 of the 54,376 rings are not valid polygons; 2080 of them +are only a zero-area spur, a wall overshooting a corner (the source wall chain crosses itself and closes a sliver lobe: +scene_000060::room_3, 2.5 x 6.5 cm) or a vertex touching the ring, and are repaired (`repair`, recorded in +meta.boundary_repaired; ring moved by median 6 cm, p90 12 cm, max 1.2 m for a zero-area spur; dropped lobes p99 +0.05 m2, max 0.24 m2, holding no placeable object centre; kept with the lobe, 3 of the 314 repaired rooms build flags +oob_objects would lose that flag). Doors/windows keep their source pose: 2 of the 8,792 in repaired rooms stood on a +dropped spur/lobe wall and now sit 5 / 12 cm outside the ring, no footprint inside it (scene_011725::room_2, +scene_002826::room_4); the other 170,616 door/window centres in polygon rooms are within 3 cm of their ring. The +other 72 fall apart into several polygons (63) or change area by >= 1% (9): boundary_type None. +Front = local -Y (front test: toilets/beds touch the wall with local +Y), same as the other Manycore exports. +Openings: room.doors / room.windows = [{wall_index, position{x,y,z} (CENTER of the opening, on the wall line), +width (along the wall), height, sill_height (windows only; doors null)}], walls[wall_index] = {start, end, thickness} +(thickness is always 0.0 in the export). Emitted as "structure" boxes width x 0.10 m x height with yaw = wall +direction; center z passes through center_z (door bottom = 0 for 99.7%, the rest are real raised openings). + +Cross-export duplicates (same Manycore design; exact floor area / boundary edges + identical pairwise distances of +floor furniture, see the g2-layout-mixed assessment): scenes holding a SpatialGen test room are dropped (SpatialGen +is eval-only), scenes matching a Structured3D / InteriorGS house take that house's group key. +""" +import math +import os + +from shapely.geometry import Polygon +from shapely.validation import explain_validity + +from fastfill.adapters.unified import _num, convert_room, iter_records, signed_area + +SOURCE = "SpatialLM" +SPATIALGEN_DUP = {f"scene_{n:06d}" for n in ( + 646, 692, 701, 713, 734, 738, 773, 792, 793, 806, 828, 832, 1569, 1585, 1591, 1599, 1601, 1610, 1612, 1758, + 4021, 4053, 7537, 7629, 7989, 8143, 8147, 8729, 8815, 11483, + # review pass: boundary-free match (>= 3 floor objects with identical dims at identical relative positions under + # one axis-aligned rigid map, >= half of the smaller set) or same floor area +-0.1 m2 with >= 70% object match + 82, 537, 644, 667, 671, 740, 745, 822, 827, 841, 843, 1579, 1606, 1608, 1624, 1626, 1725, 1745, 4027, 4099, + 4111, 4112, 4165, 5211, 6058, 6252, 7362, 7478, 7498, 7622, 7729, 7736, 9248, 9360, 9771, 10350, 10394, 11383, + 11482, 12106, + 4359)} # pre-launch review: design twin of spatialgen_scene_00039::room_0 +GROUP = {"scene_003636": "structured3d:scene_00729", "scene_007839": "structured3d:scene_02605", + "scene_009460": "structured3d:scene_01438", "scene_011931": "structured3d:scene_00402", + "scene_012118": "structured3d:scene_01556", "scene_004716": "interiorgs:0304_840560", + "scene_011878": "interiorgs:0553_840921", + # review pass: same floor area to 0.01 m2 and 4-6 floor objects with identical dims at identical relative positions + "scene_005720": "interiorgs:0785_841229", "scene_006253": "interiorgs:0536_840884", + "scene_010373": "interiorgs:0817_841276"} + + +WALL_THICKNESS = 0.10 # the export's wall thickness is 0.0 everywhere +REPAIR_TOL = 0.01 # a repaired wall ring keeps the ring's shoelace area to within 1% + + +def repair(ring): + """Invalid wall ring -> the exterior of Polygon(ring).buffer(0) when that is ONE polygon without holes whose area + is within REPAIR_TOL of the ring's shoelace area (the floor the walls enclose is then well defined: buffer(0) only + drops the spur / sliver lobe), else None (several parts, or the area changes: the room is not the ring's). + Not make_valid: it keeps a crossed corner's sliver lobe as a second polygon (one polygon for 1457 of the 2152).""" + p, a = Polygon(ring).buffer(0), abs(signed_area(ring)) + if p.geom_type == "Polygon" and not p.interiors and abs(p.area - a) < REPAIR_TOL * a: + return [list(q) for q in p.exterior.coords[:-1]] + + +def openings(room, prefix): + """Doors and windows as unified-schema furniture in the room's frame. convert_room(front_offset_deg=-90) + swaps width/length and subtracts 90 deg, so width=thickness / length=opening width / rotation=wall+90 come out + as size [opening width, thickness, height] with yaw = wall direction (long side along the wall).""" + walls = room.get("walls") or [] + out = [] + for kind in ("door", "window"): + for k, d in enumerate(room.get(kind + "s") or []): + wi = d.get("wall_index") + if not (isinstance(wi, int) and 0 <= wi < len(walls)): + continue + w = walls[wi] + ang = math.degrees(math.atan2(w["end"]["y"] - w["start"]["y"], w["end"]["x"] - w["start"]["x"])) + out.append({"furniture_category": kind, "furniture_instance_id": f"{prefix}::{kind}_{k}", "structure": True, + "furniture_position": d["position"], "furniture_rotation": {"x": 0, "y": 0, "z": ang + 90}, + "furniture_size": {"width": w.get("thickness") or WALL_THICKNESS, "length": d["width"], + "height": d["height"]}}) + return out + + +def load(root): + path = os.path.join(root, "imChuling__3D_Room_Collections/SpatialLM_exported/spatiallm.json") + for scene in iter_records(path): + sid = scene["scene_id"] + if sid in SPATIALGEN_DUP: + continue + for room in scene["rooms"]: + furn = [{**f, "furniture_category": f["furniture_category"].replace("_", " "), + "furniture_rotation": {**f["furniture_rotation"], "x": 0, "y": 0}} + for f in room["furniture"]] + furn += openings(room, f"{sid}::{room['room_id']}") + ring = room["room_boundary"] + usable = ring and len(ring) >= 3 and all(_num(v) for pt in ring for v in pt[:2]) # convert_room's test + fix = repair(ring) if usable and not Polygon(ring).is_valid else None + ir = convert_room({**room, "room_boundary": fix or ring, "furniture": furn}, source=SOURCE, + uid=f"spatiallm:{sid}::{room['room_id']}", + group=GROUP.get(sid, f"spatiallm:{sid}"), boundary_type="polygon", center_z=True, front_offset_deg=-90, + extra=lambda f: {"structure": True} if f.get("structure") else {}, + meta={"scene_id": sid, "front_known": True, "source_splits": room.get("source_splits"), + "view": "spatiallm.json"}) + if fix: # provenance: the source ring in the room frame, its defect, both areas + x0, y0 = min(x for x, _ in fix), min(y for _, y in fix) + ir["meta"]["boundary_repaired"] = { + "method": "buffer(0)", "defect": explain_validity(Polygon(ring)).split("[")[0], + "area_src": round(abs(signed_area(ring)), 4), "area": round(Polygon(fix).area, 4), + "ring_src": [[x - x0, y - y0] for x, y in ring]} + b = ir["boundary"] + if b and not Polygon(b).is_valid: # wall ring crosses/touches itself: cannot prove in-bounds + ir["boundary_type"] = None + ir["meta"]["boundary_invalid"] = True + yield ir + + +if __name__ == "__main__": + # 4 x 3 room at (10, 20); door centred on the bottom wall (+X direction), window on the right wall (+Y direction) + room = {"room_id": "r", "room_type": "bedroom", "room_height": 2.6, "furniture": [], + "room_boundary": [[10, 20], [14, 20], [14, 23], [10, 23]], + "walls": [{"wall_index": i, "start": {"x": a[0], "y": a[1], "z": 0}, "end": {"x": b[0], "y": b[1], "z": 0}, + "height": 2.6, "thickness": 0.0} + for i, (a, b) in enumerate([([10, 20], [14, 20]), ([14, 20], [14, 23]), ([14, 23], [10, 23]), + ([10, 23], [10, 20])])], + "doors": [{"wall_index": 0, "position": {"x": 12, "y": 20, "z": 1.05}, "width": 0.9, "height": 2.1, + "sill_height": None}], + "windows": [{"wall_index": 1, "position": {"x": 14, "y": 21.5, "z": 1.6}, "width": 1.2, "height": 1.4, + "sill_height": 0.9}]} + ir = convert_room({**room, "furniture": openings(room, "s::r")}, source=SOURCE, uid="t", group="g", + boundary_type="polygon", center_z=True, front_offset_deg=-90, + extra=lambda f: {"structure": True} if f.get("structure") else {}) + d, w = ir["objects"] + close = lambda a, b: all(abs(x - y) < 1e-6 for x, y in zip(a, b)) + assert d["category"] == "door" and d["structure"] and not d["tilted"], d + assert close(d["size"], [0.9, 0.10, 2.1]) and close(d["pos"], [2, 0, 0]) and abs(d["yaw"]) < 1e-9, d + assert w["category"] == "window" and w["structure"], w + assert close(w["size"], [1.2, 0.10, 1.4]) and close(w["pos"], [4, 1.5, 0.9]) and abs(w["yaw"] - math.pi / 2) < 1e-9, w + + # wall rings through load() (iter_records stubbed with one scene): a zero-area spur at the min-x side (the frame + # comes from the repaired ring), the real scene_000060::room_3 corner overshoot (2.5 x 6.5 cm lobe), two squares + # touching at a vertex (two parts), a bow-tie (shoelace 0), a loop touching the ring from inside (a hole the + # exterior would fill) and a loop inside traced the same way (shoelace 17, buffer(0) 16): the first two are + # repaired and stay "polygon", the other four get boundary_type None; a 2-point ring and a ring with a null + # coordinate reach convert_room untouched (boundary None) instead of raising in Polygon() + chair = {"furniture_category": "chair", "furniture_instance_id": "c", "furniture_rotation": {"x": None, "y": None, "z": 0}, + "furniture_position": {"x": 11, "y": 21, "z": 0.4}, "furniture_size": {"width": 0.5, "length": 0.5, "height": 0.8}} + rings = {"spur": [[10, 20], [14, 20], [14, 23], [10, 23], [10, 22], [9.5, 22], [10, 22]], + "overshoot": [[6.1625, -0.275], [6.1375, -0.275], [6.1375, 1.4025], [3.25, 1.4025], [3.25, -0.21], + [6.1625, -0.21]], + "touch": [[0, 0], [1, 0], [1, 1], [2, 1], [2, 2], [1, 2], [1, 1], [0, 1]], + "bowtie": [[0, 0], [2, 2], [2, 0], [0, 2]], + "hole": [[0, 0], [4, 0], [4, 4], [2, 4], [3, 3], [1, 3], [2, 4], [0, 4]], + "twice": [[0, 0], [4, 0], [4, 4], [2, 4], [2, 3], [1, 2], [3, 2], [2, 3], [2, 4], [0, 4]], + "short": [[0, 0], [1, 0]], "null": [[0, 0], [1, 0], [None, 1]]} + iter_records = lambda path: [{"scene_id": "s", "rooms": [{"room_id": k, "room_boundary": r, "furniture": [chair]} + for k, r in rings.items()]}] + irs = {ir["meta"]["room_id"]: ir for ir in load("")} + for k in ("spur", "overshoot"): + ir, rep = irs[k], irs[k]["meta"]["boundary_repaired"] + P = Polygon(ir["boundary"]) + assert ir["boundary_type"] == "polygon" and P.is_valid and P.bounds[:2] == (0, 0), ir + assert rep["defect"] in ("Self-intersection", "Ring Self-intersection") and abs(rep["area"] - P.area) < 1e-3, rep + assert abs(rep["area"] - rep["area_src"]) < REPAIR_TOL * rep["area_src"] and len(rep["ring_src"]) == len(rings[k]) + assert close(irs["spur"]["objects"][0]["pos"], [1, 1, 0]) and abs(Polygon(irs["spur"]["boundary"]).area - 12) < 1e-9 + assert min(x for x, _ in irs["spur"]["meta"]["boundary_repaired"]["ring_src"]) == -0.5 + assert abs(irs["overshoot"]["meta"]["boundary_repaired"]["area"] - 2.8875 * 1.6125) < 1e-4 + for k in ("touch", "bowtie", "hole", "twice"): + assert irs[k]["boundary_type"] is None and irs[k]["meta"]["boundary_invalid"], irs[k] + assert "boundary_repaired" not in irs[k]["meta"] and irs[k]["boundary"] is not None + for k in ("short", "null"): + assert irs[k]["boundary"] is None and irs[k]["boundary_type"] is None and len(irs[k]["objects"]) == 1, irs[k] + assert not {"boundary_repaired", "boundary_invalid"} & set(irs[k]["meta"]), irs[k] + print("spatiallm.py self-check ok") diff --git a/code/fastfill/adapters/structured3d.py b/code/fastfill/adapters/structured3d.py new file mode 100644 index 0000000000000000000000000000000000000000..ed84b317d4960e591269f8e4667f667996a62970 --- /dev/null +++ b/code/fastfill/adapters/structured3d.py @@ -0,0 +1,130 @@ +"""Structured3D (imChuling export, structured3d.jsonl: 22,894 rooms / 3,488 houses, official train/val/test). + +Export (kits/convert_structured3d.py): meters, Z-up, room-local (floor polygon centroid, z from floor_z); +position = bbox_3d centroid (box center); the exported size/yaw come from a heuristic axis canonicalization +(row most aligned with world X), which discards the model frame. The raw bbox_3d fields are kept per object: +raw_basis rows = box axes in world, raw_coeffs = half extents (mm) along those rows. Row 2 is the model's up; +wall-contact statistics on beds / TVs / pictures / toilets put the back on raw +Y, so front = raw -Y. +Objects whose raw row 2 is not world +Z (~31k, almost all 'unknown'/'otherprop') are tilted relative to their +model pose -> tilted=True (raw_basis kept), boxed by the upright envelope of their rotated corners. Yaw follows +raw -Y when horizontal; when that front is near vertical, use raw X and mark front_known=False. +Category 'unknown' (41%: occluded in every view, or the corrupt +perspective_full_09.zip for scenes 1600-1799) keeps its box with category "unknown": build.prep drops it as a +generic label and rejects the room only if dropped floor boxes leave a real hole (hidden_floor_obstacle). +room_type 'undefined' -> None. +Boundary = floor polygon traced from annotation_3d junctions/lines. +Doors/windows: room_context.doors/windows (same room frame, position = opening centre on the wall centreline, +~0.1 m outside the floor polygon; width, height, no direction/thickness) -> structure boxes, see `openings`. +""" +import math +import os + +import numpy as np + +from fastfill.adapters.internscenes import tilted_bbox +from fastfill.adapters.unified import convert_room, iter_records + +SOURCE = "Structured3D" + + +def _furniture(o): + B = np.array(o["raw_basis"], float) + ext = 2 * np.abs(np.array(o["raw_coeffs"], float)) / 1000 + f = {"furniture_category": o["category"], + "furniture_instance_id": str(o["instance_id"]), "furniture_position": o["position"], + "furniture_size": {"width": ext[1], "length": ext[0], "height": ext[2]}, "raw_basis": o["raw_basis"]} + if B[2, 2] < 0.999: # model up is not world up + R = np.stack([-B[1], B[0], B[2]], axis=1) # semantic front first + size, yaw, k, _ = tilted_bbox(R, ext[[1, 0, 2]]) + return {**f, "furniture_size": dict(zip(("width", "length", "height"), size)), + "furniture_rotation": {"x": None, "y": None, "z": yaw}, + **({"front_known": False} if k == 0 else {})} + front = -B[1] + return {**f, "furniture_rotation": {"x": 0, "y": 0, "z": math.degrees(math.atan2(front[1], front[0]))}} + + +OPENING_T = 0.10 # opening box thickness (the export gives none) +SAME_M = 0.05 # an object box centred this close (xy and z) to an opening, same size, is that opening's box + + +def _edge_deg(poly, x, y): + """Direction (deg) of the floor-polygon edge nearest to (x, y).""" + def dist(a, b): + dx, dy = b[0] - a[0], b[1] - a[1] + t = min(1.0, max(0.0, ((x - a[0]) * dx + (y - a[1]) * dy) / (dx * dx + dy * dy or 1))) + return math.hypot(x - a[0] - t * dx, y - a[1] - t * dy) + a, b = min(zip(poly, poly[1:] + poly[:1]), key=lambda e: dist(*e)) + return math.degrees(math.atan2(b[1] - a[1], b[0] - a[0])) + + +def openings(rc, objs): + """room_context doors/windows -> (structure furniture, objs without the boxes they replace). Box = [width along + the wall, OPENING_T, height] centred at the given position (z = centre -> bottom = sill), yaw = nearest floor + edge. ~15% of openings already carry an object box (unknown / otherstructure, same centre and size): that box is + replaced by the labelled opening instead of duplicated.""" + out, dup = [], set() + for kind in ("door", "window"): + for k, d in enumerate(rc.get(kind + "s") or []): + p, wh = d["position"], sorted([d["width"], d["height"]]) + for i, o in enumerate(objs): + q, ext = o["position"], sorted(2 * abs(c) / 1000 for c in o["raw_coeffs"])[1:] + if math.hypot(q["x"] - p["x"], q["y"] - p["y"]) < SAME_M and abs(q["z"] - p["z"]) < SAME_M \ + and all(abs(u - v) < 0.3 for u, v in zip(ext, wh)): + dup.add(i) + out.append({"furniture_category": kind, "furniture_instance_id": f"{kind}_{k}", "structure": True, + "furniture_position": p, + "furniture_rotation": {"x": 0, "y": 0, "z": _edge_deg(rc["floor_polygon"], p["x"], p["y"])}, + "furniture_size": {"width": d["width"], "length": OPENING_T, "height": d["height"]}}) + return out, [o for i, o in enumerate(objs) if i not in dup] + + +def convert(rec): + rc, prov = rec["room_context"], rec["provenance"] + ops, objs = openings(rc, rec["layout"]["floor_layout"]["objects"]) + room = {"room_id": rc["room_id"], "room_type": None if rc["room_type"] == "undefined" else rc["room_type"], + "room_boundary": rc["floor_polygon"], "room_height": rc["ceiling_height_m"], + "furniture": [_furniture(o) for o in objs] + ops} + return convert_room(room, source=SOURCE, uid=rec["sample_id"], group=f"structured3d:{prov['source_house_id']}", + subset=prov["split"], boundary_type="polygon", center_z=True, + extra=lambda f: {"structure": True} if f.get("structure") else + {**({"raw_basis": f["raw_basis"]} if f["furniture_rotation"]["x"] is None else {}), + **({"front_known": False} if f.get("front_known") is False else {})}, + meta={"front_known": True, "room_origin_house_m": rec["room_origin_house_m"], + "n_nearest_assigned": sum(o.get("assignment_method") == "nearest" for o in objs), + "n_unknown_category": sum(o["category"] == "unknown" for o in objs)}) + + +def load(root): + path = os.path.join(root, "imChuling__3D_Room_Collections/Structured3D_exported/structured3d.jsonl") + for rec in iter_records(path): + yield convert(rec) + + +if __name__ == "__main__": + # bed rotated 90 deg: raw row0 = world +Y, row1 = world -X (back on raw +Y -> world -X) -> front world +X, yaw 0 + f = _furniture({"category": "bed", "instance_id": 1, "position": {"x": 0, "y": 0, "z": 0.5}, + "raw_basis": [[0, 1, 0], [-1, 0, 0], [0, 0, 1]], "raw_coeffs": [800, 1000, 500]}) + assert abs(f["furniture_rotation"]["z"]) < 1e-9 and f["furniture_size"]["width"] == 2.0, f + f = _furniture({"category": "unknown", "instance_id": 2, "position": {"x": 0, "y": 0, "z": 1}, + "raw_basis": [[0, 0, -1], [0, 1, 0], [1, 0, 0]], "raw_coeffs": [1, 1, 1]}) + assert f["furniture_category"] == "unknown" and f["furniture_rotation"]["x"] is None + # 4 x 3 room; door on the bottom wall already boxed as 'otherstructure' (replaced, not duplicated), window on the + # right wall (x = 4) 0.1 m outside the floor, centre z 1.5, height 1.4 -> sill 0.8 + rec = {"sample_id": "t", "provenance": {"source_house_id": 0, "split": "train"}, "room_origin_house_m": [0, 0, 0], + "room_context": {"room_id": "r", "room_type": "undefined", "ceiling_height_m": 2.8, + "floor_polygon": [[0, 0], [4, 0], [4, 3], [0, 3]], + "doors": [{"position": {"x": 1, "y": -0.06, "z": 1.1}, "width": 0.7, "height": 2.2}], + "windows": [{"position": {"x": 4.1, "y": 1.5, "z": 1.5}, "width": 1.6, "height": 1.4}]}, + "layout": {"floor_layout": {"objects": [ + {"category": "otherstructure", "instance_id": 9, "position": {"x": 1, "y": -0.06, "z": 1.1}, + "raw_basis": [[1, 0, 0], [0, 1, 0], [0, 0, 1]], "raw_coeffs": [350, 60, 1100]}]}}} + ir = convert(rec) + door, win = ir["objects"] + close = lambda a, b: all(abs(x - y) < 1e-6 for x, y in zip(a, b)) + assert door["category"] == "door" and door["structure"] and close(door["pos"], [1, -0.06, 0]), door + assert close(door["size"], [0.7, OPENING_T, 2.2]) and abs(door["yaw"]) < 1e-9, door + assert win["category"] == "window" and close(win["pos"], [4.1, 1.5, 0.8]) and close(win["size"], [1.6, OPENING_T, 1.4]) + assert abs(win["yaw"] - math.pi / 2) < 1e-9 and not win["tilted"], win + from fastfill.anchors import fixed_geometry + assert {o["category"] for o in fixed_geometry(ir)} == {"door", "window"} + print("structured3d.py self-check ok") diff --git a/code/fastfill/adapters/unified.py b/code/fastfill/adapters/unified.py new file mode 100644 index 0000000000000000000000000000000000000000..9bd84207929e92762bfa6f6e0d0bb0cba049ca6c --- /dev/null +++ b/code/fastfill/adapters/unified.py @@ -0,0 +1,112 @@ +"""Shared reader for the common export schema used by most collections: +scene{scene_id, provenance, rooms[{room_id, room_type, room_boundary, room_boundary_type, room_height, +furniture[{furniture_category, furniture_instance_id, furniture_position{x,y,z}, +furniture_rotation{x,y,z} (deg), furniture_size{width,length,height}, source_fields}]}]}. + +Conventions differ per source (box center vs bottom, which local axis is the front, boundary quality), +so each adapter passes them explicitly to `convert_room`. +""" +import json +import math + +import ijson + + +def iter_records(path): + """Scene records from a .jsonl or a top-level JSON array, streamed (some files are >8 GB).""" + if path.endswith(".jsonl"): + with open(path) as f: + for line in f: + if line.strip(): + yield json.loads(line) + else: + with open(path, "rb") as f: + yield from ijson.items(f, "item", use_float=True) + + +def signed_area(poly): + return sum(x0 * y1 - x1 * y0 for (x0, y0), (x1, y1) in zip(poly, poly[1:] + poly[:1])) / 2 + + +def _num(v): + return isinstance(v, (int, float)) and math.isfinite(v) + + +def convert_room(room, *, source, uid, group, subset=None, boundary_type=None, center_z=False, + front_offset_deg=0.0, size_keys=("width", "length", "height"), extra=None, meta=None): + """One exported room -> IR room. + + boundary_type IR boundary type for this source ("polygon" | "hull" | "proxy" | None) + center_z True if furniture_position.z is the box center (converted to bottom) + front_offset_deg angle of the object's semantic front in its exported local frame, measured CCW from + local +X; the IR frame is rotated so the front becomes local +X (sizes swap for +-90) + size_keys keys of furniture_size along exported local X, Y, Z + extra(f) optional per-furniture dict merged into the IR object (anchor, parent, desc, ...) + Objects with a missing category/size/position/yaw are counted in meta.n_incomplete; tilted objects + (rotation x/y null or non-zero) get tilted=True. fastfill.build rejects rooms with incomplete objects + and rooms whose in-scope objects are tilted. + """ + objs, n_tilted, n_incomplete = [], 0, 0 + swap = round(front_offset_deg) % 180 == 90 + for k, f in enumerate(room.get("furniture") or []): + s, p, r = f.get("furniture_size"), f.get("furniture_position"), f.get("furniture_rotation") + if not (f.get("furniture_category") and s and p and r): + n_incomplete += 1 + continue + size = [s.get(key) for key in size_keys] + pos = [p.get("x"), p.get("y"), p.get("z")] + if not all(_num(v) for v in size + pos) or not _num(r.get("z")): + n_incomplete += 1 + continue + tilted = r.get("x") != 0 or r.get("y") != 0 + n_tilted += tilted + if center_z: + pos[2] -= size[2] / 2 + if swap: + size[0], size[1] = size[1], size[0] + o = {"id": f.get("furniture_instance_id") or f"obj_{k}", "category": f["furniture_category"], + "size": size, "pos": pos, "yaw": math.radians(r["z"] + front_offset_deg) % (2 * math.pi), + "anchor": None, "parent": None, "tilted": tilted} + if extra: + o.update(extra(f)) + objs.append(o) + + boundary = room.get("room_boundary") + if boundary and len(boundary) >= 3 and all(_num(v) for pt in boundary for v in pt[:2]): + boundary = [[float(pt[0]), float(pt[1])] for pt in boundary] + if signed_area(boundary) < 0: + boundary = boundary[::-1] + x0, y0 = min(p[0] for p in boundary), min(p[1] for p in boundary) + boundary = [[x - x0, y - y0] for x, y in boundary] + else: + boundary, boundary_type = None, None + x0 = min((o["pos"][0] for o in objs), default=0.0) + y0 = min((o["pos"][1] for o in objs), default=0.0) + for o in objs: + o["pos"] = [o["pos"][0] - x0, o["pos"][1] - y0, o["pos"][2]] + + height = room.get("room_height") + return {"uid": uid, "source": source, "subset": subset, "group": group, + "room_type": room.get("room_type"), "boundary": boundary, "boundary_type": boundary_type, + "height": height if _num(height) else None, "objects": objs, + "meta": {"n_tilted": n_tilted, "n_incomplete": n_incomplete, "room_id": room.get("room_id"), + **(meta or {})}} + + +if __name__ == "__main__": + room = {"room_id": "r", "room_type": "bedroom", "room_boundary": [[1, 1], [1, 4], [5, 4], [5, 1]], + "furniture": [ + {"furniture_category": "bed", "furniture_instance_id": "b", + "furniture_position": {"x": 2, "y": 2, "z": 0.5}, "furniture_rotation": {"x": 0, "y": 0, "z": 0}, + "furniture_size": {"width": 2.0, "length": 1.6, "height": 1.0}}, + {"furniture_category": "lamp", "furniture_position": {"x": 2, "y": 2, "z": 0.5}, + "furniture_rotation": {"x": None, "y": None, "z": 10}, "furniture_size": {"width": 1, "length": 1, "height": 1}}, + {"furniture_category": None}]} + ir = convert_room(room, source="t", uid="t::r", group="g", boundary_type="polygon", center_z=True, + front_offset_deg=-90) + assert [0.0, 0.0] in ir["boundary"] and signed_area(ir["boundary"]) > 0 + bed = ir["objects"][0] + assert bed["pos"] == [1.0, 1.0, 0.0] and bed["size"] == [1.6, 2.0, 1.0] + assert abs(bed["yaw"] - math.radians(270)) < 1e-9 + assert ir["meta"]["n_tilted"] == 1 and ir["meta"]["n_incomplete"] == 1 + print("unified.py self-check ok") diff --git a/code/fastfill/anchors.py b/code/fastfill/anchors.py new file mode 100644 index 0000000000000000000000000000000000000000..03d9d267781b9b5b6bd957029d3099166107bf34 --- /dev/null +++ b/code/fastfill/anchors.py @@ -0,0 +1,239 @@ +"""Object scope for training: drop structure, infer support anchors, keep a support-closed subset. + +anchor = "floor" | "object" | "wall" | "ceiling" | "fixed". Adapters set anchor/parent from source evidence when the +source has it (placement flags, explicit parent ids); otherwise `infer_anchors` derives them from boxes: + ceiling hanging categories (pendant, chandelier, ...) whatever their z or source anchor (one at z=0 is + misplaced), or top within 10 cm of the room height above 1 m + floor bottom at most FLOOR_Z = 10 cm above z=0 (sunk boxes too, so build.prep can drop and count them), or + floor furniture (bed, table, sofa, ...) at most FURNITURE_FLOOR_Z = 50 cm up with nothing under it + (annotation z-noise, boxes that start above the legs; build.prep snaps them to the floor) + object bottom within 5 cm of the top of a floor/object box whose footprint contains its center; + seats, beds and rugs are never parents (bbox top = backrest / headboard; a rug is not a surface) + wall wall art (painting, mirror, ...) with no support below, or anything else elevated without one + fixed floor furniture that can hang (cabinet, shelf, vanity...) floating <= 50 cm with nothing below in a + designer (polygon) room: a wall-hung unit, out of scope to place, shown to the model as a fixed box +Box-level contact is weak evidence of support; build.py counts evidence and inferred supports separately. +`fixed_geometry` picks the structure boxes the model is told about (never placed): doors, windows, columns, stairs, +fireplaces, radiators, partitions, low beams... below FIXED_MAX_Z; surface finishes, sockets/drains, curtains and +slabs covering most of the room are left out, and a "wall" box only counts when it stands inside the room. +""" +import math +import re + +from shapely.geometry import Point, Polygon + +from fastfill.scene import footprint, head, norm_cat + +FLOOR_Z = 0.10 # bottoms up to 10 cm above the floor count as standing on it (build.prep snaps them to z=0) +# Strong architectural words mark structure anywhere in the name; floor/wall/ceiling/column are usually modifiers +# (plant_floor, floor lamp, wall_shelving_unit, locker_column) and mark structure only in purely architectural +# names ("wall", "Wall design", "floor mold"). +STRUCTURE = re.compile(r"\b(door|window|curtain|drape|blind|beam|pillar|stair|staircase|stairway|stairwell|railing|rail|" + r"handrail|banister|guardrail|balustrade|baseboard|radiator|opening|doorframe|partition|fireplace|" + r"chimney|otherstructure|structure|doorway|arch|archway|bulkhead|niche|fence|step|ramp|elevator|duct|" + r"drain|pipe|socket|outlet|tile)s?\b") +# furniture whose name borrows a structure word: fireplace_console, pillar_candle, clothes rail, portable_partition_panel +NOT_STRUCTURE = re.compile(r"\b(lamp|light|fan|cabinet|shelf|mat|rug|towel|chair|table|console|stool|candle|figurine|box|" + r"plant|planter|basket|display|station|clothes|freestanding|portable|ladder|bench|sofa|couch|" + r"bed|wardrobe|desk|seat|bookcase|bookshelf|dresser|sideboard|ottoman)s?\b") +SURFACE = r"(wall|floor|ceiling|column)s?" +ARCH = (r"(panel|paneling|design|combination|mold|molding|moulding|background|trim|skirting|board|tile|cladding|covering|" + r"decoration|sticker|styling|decorative|decor|structural|support|concrete|low|half|suspended)s?") +HANGING = re.compile(r"\b(pendant|chandelier|ceiling|downlight|hanging)s?\b") +NOT_HANGING = re.compile(r"\b(chair|stool|seat|swing)\b") # a hanging chair stands on its own frame +# soft covers draped over a bed or sofa (their boxes cover the bed): not objects to place, not floor obstacles +SOFT_COVERS = {"bedding", "comforter", "blanket", "quilt", "duvet", "bedspread", "bedcover", "coverlet"} +# floor furniture whose box bottom sits up to FURNITURE_FLOOR_Z high with nothing under it stands on the floor +# (annotation z-noise, boxes that start above the legs); without this it fell through to 'wall' and vanished +FLOOR_FURNITURE = {"bed", "sofa", "couch", "loveseat", "sectional", "table", "desk", "chair", "armchair", "stool", "bench", + "ottoman", "cabinet", "wardrobe", "dresser", "bookcase", "bookshelf", "sideboard", "nightstand", + "fridge", "refrigerator", "oven", "stove", "dishwasher", "washer", "dryer", "piano", "tub", + "bathtub", "toilet", "console", "stand", "chest", "armoire", "shelf", "cupboard"} +FURNITURE_FLOOR_Z = 0.50 # floating up to here with nothing below: a floor piece (snapped to z=0 by build.prep) +FURNITURE_SUNK_Z = 0.30 # any floor box sunk up to here: same (deeper boxes belong to a lower storey) +# furniture that cannot hang on a wall: floating with nothing below it is on the floor with a noisy z anywhere +GROUND_ONLY = {"bed", "sofa", "couch", "loveseat", "sectional", "chair", "armchair", "table", "desk", "fridge", + "refrigerator", "oven", "stove", "dishwasher", "stool", "bench", "ottoman", "piano", "tub", "bathtub", + "washer", "dryer", "wardrobe", "armoire", "bookcase", "dresser"} +SOFT_HANGINGS = {"curtain", "curtains", "drape", "drapes", "blind", "blinds", "shade", "shades", "screening", "valance"} +# structure that has no footprint of its own: finishes, fixtures, the room's own slabs +FINISH = {"floor", "ceiling", "tile", "tiles", "baseboard", "skirting", "trim", "mold", "molding", "moulding", "panel", + "paneling", "board", "boards", "cladding", "covering", "socket", "outlet", "drain", "pipe", "pipes", "design", + "decoration", "sticker", "background", "wallpaper"} +FIXED_MAX_Z = 1.5 # structure whose bottom is higher does not shape the floor layout (ceiling beams, ducts) +DIVIDERS = {"partition", "divider", "screen"} # a "partition panel" is an obstacle, not a wall finish +# scans: box z is noisy, so floating floor furniture is on the floor (not a wall-hung unit), whatever the boundary type +SCANNED = {"Scan2CAD", "MultiScan", "InternScenes_arkit", "InternScenes_scannet", "InternScenes_mp3d", + "InternScenes_3rscan"} + + +def hanging(category): + nc = norm_cat(category) + return bool(HANGING.search(nc)) and not NOT_HANGING.search(nc) + + +WALL_ART = re.compile(r"\b(painting|picture|poster|artwork|art|mirror|whiteboard|blackboard|frame)s?\b") +FLAT = {"rug", "carpet", "mat", "doormat", "bathmat", "placemat", "playmat", "floormat", "yogamat", "exercisemat", + "fitnessmat"} +FLAT |= {w + "s" for w in FLAT} +NO_TOP = {"sofa", "couch", "loveseat", "sectional", "bed", "armchair", "chair", "bench"} | FLAT # not a support surface + + +def is_structure(category): + c = re.sub(r"[^a-z]+", " ", (category or "").lower()) + if NOT_STRUCTURE.search(c): + return False + return bool(STRUCTURE.search(c)) or (bool(re.search(rf"\b{SURFACE}\b", c)) + and not re.sub(rf"\b({SURFACE}|{ARCH})\b", " ", c).strip()) + + +def infer_anchors(room, tol=0.05): + objs = room["objects"] + fps = {o["id"]: Polygon(footprint(o)) for o in objs} + height = room.get("height") + hull = room.get("boundary_type") == "hull" or room.get("source") in SCANNED + for o in sorted(objs, key=lambda o: o["pos"][2]): + if hanging(o["category"]): # before the floor rule and any source anchor: a chandelier at z=0 is misplaced + o["anchor"] = "ceiling" + continue + if o.get("anchor"): + continue + z, nc = o["pos"][2], norm_cat(o["category"]) + if z <= FLOOR_Z: # one-sided: sunk boxes stay floor so build.prep drops and counts them + o["anchor"] = "floor" + continue + c = Point(o["pos"][0], o["pos"][1]) + below = [p for p in objs if p is not o and p.get("anchor") in ("floor", "object") and head(p) not in NO_TOP + and abs(p["pos"][2] + p["size"][2] - z) < tol and fps[p["id"]].contains(c)] + if below: # a frame standing on a dresser is on it; art above a sofa is not (NO_TOP) + o["anchor"], o["parent"] = "object", max(below, key=lambda p: p["pos"][2] + p["size"][2])["id"] + o["anchor_inferred"] = True + elif WALL_ART.search(nc): + o["anchor"] = "wall" + elif z <= FURNITURE_FLOOR_Z and head(o) in FLOOR_FURNITURE: + # nothing under it: on the floor with a noisy z (scans; anything that cannot hang), or a wall-hung + # unit in a designer room (vanity, floating TV console): shown as fixed, not placed + o["anchor"] = "floor" if hull or head(o) in GROUND_ONLY else "fixed" + elif height and z + o["size"][2] > height - 2 * tol and z > 1.0: + o["anchor"] = "ceiling" + else: + o["anchor"] = "wall" + return room + + +def fixed_geometry(room, band=0.10, intrusion=0.05): + """Structure boxes to show the model as immovable (see module doc). Returns copies with tilted/anchor cleared. + Which of them survive next to the reference layout (covered, outside, slabs) is build.prep's decision.""" + out = [] + shell = Polygon(room["boundary"]).buffer(0) if room.get("boundary") else None + inner = shell.buffer(-band) if shell is not None else None + hull = room.get("boundary_type") == "hull" + for o in room["objects"]: + if not structural(o): + continue + words, h = set(norm_cat(o["category"]).split()), head(o) + finish = (h in FINISH or h.rstrip("s") in FINISH) and not words & DIVIDERS + if finish or words & SOFT_HANGINGS or o["pos"][2] >= FIXED_MAX_Z: + continue + fp = Polygon(footprint(o)) + if shell is not None and fp.area >= 0.8 * shell.area: # a slab under another name + continue + # a "wall" box is the room's own wall unless it stands inside a polygon room (a partition); in a hull room + # the walls stand inside the hull by construction, so none of them is a partition + if h == "wall" and (hull or inner is None or fp.intersection(inner).area < intrusion): + continue + out.append({k: v for k, v in o.items() if k not in ("anchor", "parent", "tilted", "anchor_inferred")}) + return out + + +def structural(o): + """Source-flagged architecture, or a structure name on something that does not stand on another object (a + power 'outlet' strip or a lever 'arch' file on a desk is an object, not architecture).""" + return bool(o.get("structure")) or (is_structure(o["category"]) and not o.get("parent")) + + +def annotate(room): + """Structure and soft covers removed, anchors inferred: every remaining box with anchor/parent set.""" + objs = [dict(o) for o in room["objects"] if not (structural(o) or head(o) in SOFT_COVERS)] + return infer_anchors({**room, "objects": objs})["objects"] + + +def closed(objs, keep=("floor", "object")): + """Objects whose anchor is in `keep` and whose support chain is entirely kept, in input order.""" + kept, ids = [], set() + for o in sorted(objs, key=lambda o: o["pos"][2]): # parents sit lower than children + if o["anchor"] in keep and (o.get("parent") is None or o["parent"] in ids): + kept.append(o) + ids.add(o["id"]) + order = {o["id"]: k for k, o in enumerate(objs)} + return sorted(kept, key=lambda o: order[o["id"]]) + + +def scope_objects(room, keep=("floor", "object")): + """Structure removed, anchors inferred, objects outside `keep` dropped together with their dependants.""" + return {**room, "objects": closed(annotate(room), keep)} + + +if __name__ == "__main__": + room = {"height": 2.6, "objects": [ + {"id": "desk", "category": "desk", "size": [1.2, .6, .75], "pos": [1, 1, 0], "yaw": 0}, + {"id": "lamp", "category": "table lamp", "size": [.2, .2, .4], "pos": [1.2, 1, .75], "yaw": 0}, + {"id": "pic", "category": "painting", "size": [.8, .05, .6], "pos": [1, 0, 1.4], "yaw": 0}, + {"id": "cl", "category": "pendant lamp", "size": [.5, .5, .3], "pos": [2, 2, 2.0], "yaw": 0}, + {"id": "sofa", "category": "sofa", "size": [2, .9, .8], "pos": [3, 3, 0], "yaw": 0}, + {"id": "art", "category": "picture frame", "size": [.6, .05, .5], "pos": [3, 3, .8], "yaw": 0}, + {"id": "vase", "category": "vase", "size": [.1, .1, .2], "pos": [1, 0, 2.0], "yaw": 0, "anchor": "object", "parent": "pic"}, + {"id": "win", "category": "window", "size": [1, .1, 1.2], "pos": [3, 0, .9], "yaw": 0}]} + s = scope_objects(room) + assert [o["id"] for o in s["objects"]] == ["desk", "lamp", "sofa"], s + assert s["objects"][1]["parent"] == "desk" and s["objects"][1]["anchor"] == "object" + a = {o["id"]: o["anchor"] for o in infer_anchors({**room, "objects": [dict(o) for o in room["objects"]]})["objects"]} + assert a["pic"] == "wall" and a["cl"] == "ceiling" and a["art"] == "wall", a # art above a sofa is not "on" it + assert is_structure("window") and is_structure("door(screen door, screen)") and not is_structure("floor lamp") + for c in ("wall", "floor", "Wall design", "ceiling combination", "floor mold", "otherstructure", "room_partition", + "radiator", "support_column", "vertical core stair 1", "door(screen door, screen)"): + assert is_structure(c), c + for c in ("plant_floor", "floor_plant_large", "locker_column", "floor_bookshelf", "bookshelf_wall_unit", + "floor-standing air purifier", "wall_hung_toilet", "floor lamp", "wall_cabinet", "ceiling fan", "wall_mirror"): + assert not is_structure(c), c + sofa = {"id": "s", "category": "sofa", "size": [2, .9, .8], "pos": [1, 1, 0.07], "yaw": 0} + rug = {"id": "r", "category": "rug", "size": [3, 2, .02], "pos": [1, 1, 0], "yaw": 0} + got = {o["id"]: (o["anchor"], o.get("parent")) for o in infer_anchors({"objects": [dict(sofa), dict(rug)]})["objects"]} + assert got == {"s": ("floor", None), "r": ("floor", None)}, got # 7 cm up = standing, not on the rug + fx = {"boundary": [[0, 0], [5, 0], [5, 4], [0, 4]], "objects": [ + {"id": "d", "category": "door", "size": [.9, .1, 2.1], "pos": [2, 0, 0], "yaw": 0}, + {"id": "w", "category": "window", "size": [1.2, .1, 1.3], "pos": [4.98, 2, .9], "yaw": math.pi / 2}, + {"id": "c", "category": "support_column", "size": [.3, .3, 2.8], "pos": [2.5, 2, 0], "yaw": 0, "structure": True}, + {"id": "b", "category": "beam", "size": [5, .3, .3], "pos": [2.5, 2, 2.5], "yaw": 0}, + {"id": "f", "category": "floor", "size": [5, 4, .02], "pos": [2.5, 2, 0], "yaw": 0}, + {"id": "k", "category": "curtain", "size": [1.5, .1, 2.4], "pos": [4.95, 2, 0], "yaw": 0}, + {"id": "s", "category": "socket", "size": [.08, .02, .08], "pos": [1, 0.01, .3], "yaw": 0}, + {"id": "w1", "category": "wall", "size": [4, .1, 2.8], "pos": [2.5, 0.05, 0], "yaw": 0}, + {"id": "w2", "category": "wall", "size": [2, .1, 2.8], "pos": [2.5, 2, 0], "yaw": 0}, + {"id": "p", "category": "wall panels", "size": [3, .05, 2.8], "pos": [1.5, 0.02, 0], "yaw": 0}, + {"id": "sc", "category": "window screening", "size": [1.5, .05, 2.4], "pos": [2, 3.98, 0], "yaw": 0}, + {"id": "t", "category": "table", "size": [1.2, .6, .75], "pos": [1, 1, 0], "yaw": 0}]} + assert [o["id"] for o in fixed_geometry(fx)] == ["d", "w", "c", "w2"], fixed_geometry(fx) + assert [o["id"] for o in fixed_geometry({**fx, "boundary_type": "hull"})] == ["d", "w", "c"] # hull: walls are not partitions + assert not is_structure("step ladder") and is_structure("styling column") and is_structure("otherstructure") + assert is_structure("elevator shaft") and is_structure("banister") and not is_structure("floor lamp") + assert not is_structure("elevator_lobby_bench") and is_structure("stair_railing") and is_structure("bedroom_door") + panel = {"boundary": fx["boundary"], "objects": [{"id": "pp", "category": "low_partition_panel", "size": [.3, 1, 1.7], + "pos": [2.5, 2, 0], "yaw": 0}, {"id": "wp", "category": "wall panels", + "size": [3, .05, 2.8], "pos": [1.5, 0.02, 0], "yaw": 0}]} + assert [o["id"] for o in fixed_geometry(panel)] == ["pp"] # a partition panel is fixed, a wall panel is not + # a frame standing on a dresser is on it; a wall-hung cabinet floats -> fixed in a polygon room, floor in a hull + dr = {"id": "dr", "category": "dresser", "size": [1, .5, .8], "pos": [1, 1, 0], "yaw": 0} + fr = {"id": "fr", "category": "picture frame", "size": [.2, .05, .25], "pos": [1, 1, .8], "yaw": 0} + cb = {"id": "cb", "category": "bathroom cabinet", "size": [.6, .4, .5], "pos": [3, 3, .35], "yaw": 0} + bd = {"id": "bd", "category": "bed", "size": [2, 1.6, .5], "pos": [3, 1, .35], "yaw": 0} + got = {o["id"]: (o["anchor"], o.get("parent")) for o in annotate({"boundary_type": "polygon", "objects": [dr, fr, cb, bd]})} + assert got == {"dr": ("floor", None), "fr": ("object", "dr"), "cb": ("fixed", None), "bd": ("floor", None)}, got + got = {o["id"]: o["anchor"] for o in annotate({"boundary_type": "hull", "objects": [dict(cb)]})} + assert got == {"cb": "floor"}, got + assert annotate({"source": "Scan2CAD", "boundary_type": "polygon", "objects": [dict(cb)]})[0]["anchor"] == "floor" + strip = {"id": "ps", "category": "power_outlet_strip", "size": [.3, .06, .04], "pos": [1, 1, .8], "yaw": 0, + "anchor": "object", "parent": "dr"} + assert [o["id"] for o in annotate({"objects": [dict(dr), strip]})] == ["dr", "ps"] and not fixed_geometry( + {"boundary": fx["boundary"], "objects": [dict(dr), strip]}) + print("anchors.py self-check ok") diff --git a/code/fastfill/build.py b/code/fastfill/build.py new file mode 100644 index 0000000000000000000000000000000000000000..c42753a7c63200b84a94ab26b8a484657cc8f16a --- /dev/null +++ b/code/fastfill/build.py @@ -0,0 +1,623 @@ +"""IR rooms -> filtered, deduplicated, leakage-safe SFT files (spec: FastFill I/O design review, 2026-09-24). + + python -m fastfill.build --ir /Volumes/harddisk/fastfill_ir --out data/fastfill_v1 [--sources ...] [--constraint_frac 0.3] + +Per room: prep (reject or clean, see `prep`; structure and unlabelled floor boxes become `fixed` context, never +placed) -> cross-source dedup by group -> split by group/content +-> train: rot90 augmentation -> canonical order/ids -> optional constraints (S2) -> field dropout -> messages + dev/test: canonical -> messages (no augmentation), IR written to {dev,test}_rooms.jsonl for fastfill.evaluate, + test_constrained_rooms.jsonl adds 1-4 GT-true constraints per test room (rooms without any are skipped). +Eval-only sources (meta.eval_only) go to test with their whole split component (split.assign_splits). +stats.json counts every rejection reason, quality flag, dropped object and inferred vs evidence support per source. +Rejections are only what makes a training target impossible (no verified front, missing boxes, bad numbers, no +boundary, no placeable object); everything else (reference objects through walls, same-category overlaps, tiny rooms, +unexplained dropped floor area, single-object rooms, snapped or tilted boxes) is a FLAG written next to the sample. +""" +import argparse +import collections +import hashlib +import json +import math +import os +import random +import re +import tempfile +from contextlib import ExitStack +from pathlib import Path + +import numpy as np +import shapely +from shapely import STRtree +from shapely.geometry import LineString, Point, Polygon +from shapely.ops import unary_union + +from fastfill.anchors import (DIVIDERS, FINISH, FLAT, FLOOR_Z, FURNITURE_SUNK_Z, SOFT_COVERS, SOFT_HANGINGS, WALL_ART, + annotate, closed, fixed_geometry, hanging, is_structure) +from fastfill.scene import (GENERIC, canonical, clean_boundary, footprint, head, messages, norm_cat, num, rot90, + shown_size, slug) +from fastfill.split import assign_splits +from fastfill.validate import check, holds, support_ok + +# dropped boxes that do not block the floor: openings and wall/floor finishes (plant_floor -> plant is not exempt) +FLUSH = re.compile(r"^(doors?|doorframes?|windows?|curtains?|drapes?|blinds?|frames?|openings?|baseboards?|rails?|" + r"railings?|radiators?|beams?|shades?|screening|designs?|molds?|moldings?|panels?|paneling|combination|" + r"walls?|floors?|ceilings?|skirting|trim|tiles?|boards?)$") + + +WALL_BAND_M = 0.10 # a dropped box within this band of the boundary keeps to the walls +THICK_WALL_M = 0.30 # structure (doors, windows) may stand this far outside the floor polygon, inside the wall +MIN_AREA_M2 = 0.5 # a floor smaller than this holds no furniture: a broken (mm-scale) boundary, rejected +INTRUSION_M2 = 0.05 # ... unless more than this much of it stands further inside the room +# GT duplicate rule: soft/decor heads, lamps and composites are exempt +SOFT_HEADS = {"plant", "plants", "flower", "flowers", "pillow", "pillows", "cushion", "towel", "book", "books", + "ornament", "decoration", "vase", "toy", "toys", "clothes", "bag", "lamp", "light"} +COMPOSITE = re.compile(r"\b(combination|l shaped|composite|kids)\b") + + +def _words(category): + return re.sub(r"[^a-z]+", " ", (category or "").lower()).split() + + +def _flush(o, inner, polygon): + """A dropped floor box that leaves no hole: a flat covering, a soft cover or curtain, or a structural + opening/finish ('bed frame' is not). In a polygon room (the boundary is the walls) such a box must also keep to + the walls, under INTRUSION_M2 inside `inner` (the room shrunk by WALL_BAND_M): an open door leaf, a shower + enclosure or a 'wall' box standing in the room is an obstacle whatever its label. Source-flagged architecture + (stairs/pillars are shown as fixed boxes when they qualify) needs only the geometry there. In a hull room (a scan's + floor hull, walls stand inside it) the name decides, flagged boxes may also pass by geometry.""" + w = _words(o["category"]) + if head(o) in FLAT or head(o) in SOFT_COVERS or SOFT_HANGINGS & set(w) or \ + ((head(o) in FINISH or head(o).rstrip("s") in FINISH) and not DIVIDERS & set(w)): + return True + stays = Polygon(footprint(o)).intersection(inner).area < INTRUSION_M2 + if o.get("structure"): + return stays if polygon else \ + (any(FLUSH.match(x) for x in w) and not {"partition", "screen", "clothes"} & set(w)) or stays + if not (polygon is False or stays): + return False + while len(w) > 1 and w[-1] in ("floor", "wall", "ceiling"): # placement suffix: plant_floor, bookshelf_wall + w.pop() + return bool(w) and is_structure(o["category"]) and not {"partition", "screen", "clothes"} & set(w) \ + and bool(FLUSH.match(w[-1])) + + +FRONT = {"chair", "armchair", "seat", "sofa", "couch", "loveseat", "bed", "desk", "tv", "television", "monitor", "toilet", + "piano"} +PRIORITY = ["on", "between", "faces", "against_wall", "near"] # availability in rooms, rarest first + + +def _drop_with_dependants(objs, bad): + bad = set(bad) + while True: + more = {o["id"] for o in objs if o.get("parent") in bad} - bad + if not more: + return [o for o in objs if o["id"] not in bad] + bad |= more + + +FLAT_Z = 0.15 # a box lower than this that is not placed is a floor covering, not an obstacle + + +def _finite(o): + return all(isinstance(o.get(k), (list, tuple)) and len(o[k]) == 3 for k in ("size", "pos")) and \ + all(_finite_number(v) for v in list(o["size"]) + list(o["pos"]) + [o.get("yaw")]) + + +def _finite_number(value): + if not isinstance(value, (int, float)) or isinstance(value, bool): + return False + try: + return math.isfinite(value) + except OverflowError: + return False + + +def _room_geometry_error(room): + boundary = room.get("boundary") + if not isinstance(boundary, (list, tuple)) or len(boundary) < 3 or \ + any(not isinstance(p, (list, tuple)) or len(p) != 2 for p in boundary): + return "boundary_shape" + if not all(_finite_number(v) for p in boundary for v in p): + return "bad_numbers" + if len({tuple(p) for p in boundary}) < 3: + return "boundary_shape" + height = room.get("height") + if height is not None and (not _finite_number(height) or height <= 0): + return "bad_numbers" + return None + + +def _keep_fixed(fixed, objs, shell, cover=0.5, slab=0.5): + """Fixed boxes worth showing next to the reference layout: touching the room, not a slab; a box whose top is + below FLAT_Z is a covering or trim, not an obstacle; a box sunk below the floor is cut at the floor. A NON-structure + box (unlabelled, wall-anchored, wall-hung, no-front) mostly covered by a placed object is that object's second + annotation and is dropped; a structure box (column, stair, door, window...) is real whatever stands on it, so it + is always kept (a reference layout through it shows up as a GT fixed collision).""" + kept = [(Polygon(footprint(o)), o["pos"][2], o["pos"][2] + o["size"][2]) for o in objs] + near = shell.buffer(WALL_BAND_M) + wall = shell.buffer(THICK_WALL_M) # a door or window sits in the wall: up to half a thick wall outside the floor + out = [] + for f in fixed: + z0, z1 = f["pos"][2], f["pos"][2] + f["size"][2] + if z1 < FLAT_Z: + continue + if z0 < -FLOOR_Z: + f, z0 = {**f, "z_src": z0, "pos": [f["pos"][0], f["pos"][1], 0.0], "size": [f["size"][0], f["size"][1], z1]}, 0.0 + fp = Polygon(footprint(f)) + if fp.area <= 0 or fp.area >= slab * shell.area or \ + not fp.intersects(wall if f.get("fixed_kind") == "structure" else near): + continue + over = [p for p, lo, hi in kept if min(z1, hi) - max(z0, lo) > 0.05] + if f.get("fixed_kind") != "structure" and over and unary_union(over).intersection(fp).area >= cover * fp.area: + continue + out.append(f) + return out + + +def prep(room, a): + """IR room -> (clean room, None) or (None, reject reason). The clean room carries `fixed`: boxes the model is + told about with their pose and never places (each tagged fixed_kind for stats): + structure doors, windows, columns, stairs... (anchors.fixed_geometry) + generic unlabelled boxes ('unknown', 'otherprop', 'object') standing on the floor + wall wall-anchored items whose box stands on the floor (a wall cabinet run from z=0) + floating wall-hung floor furniture in designer rooms (anchor "fixed" in anchors.infer_anchors) + no_front placeable boxes whose front the source does not give (InteriorGS custom cabinetry): the answer needs + every object's yaw, so they are shown as built-in instead of rejecting the room (flag no_front_fixed) + all of them filtered by _keep_fixed. Tilted boxes are kept as upright boxes with their yaw; floor boxes sunk + up to 30 cm and floor furniture floating up to 50 cm with nothing below (scans, or anything that cannot hang) + are snapped to the floor; wall / ceiling / inside-cabinet items are dropped without rejecting the room.""" + meta = room.get("meta", {}) + if meta.get("front_known") is not True: + return None, "front_unknown" + if meta.get("n_incomplete"): # objects without a box would be invisible obstacles + return None, "incomplete_objects" + if len({o["id"] for o in room["objects"]}) != len(room["objects"]): + return None, "duplicate_source_ids" + if room.get("boundary_type") not in a.boundary_types or not room.get("boundary"): + return None, "boundary" + if reason := _room_geometry_error(room): + return None, reason + if not all(_finite(o) for o in room["objects"]): + return None, "bad_numbers" + fixed = [{**o, "fixed_kind": "structure"} for o in fixed_geometry(room)] + allobj = annotate(room) # structure and soft covers out, anchors inferred + objs = closed(allobj, a.source_anchors.get(room["source"], a.anchors)) + generic = {o["id"] for o in objs if GENERIC.match(norm_cat(o["category"])) or not norm_cat(o["category"])} + sunk = {o["id"] for o in objs if not o.get("parent") and o["pos"][2] < -FURNITURE_SUNK_Z} + objs = _drop_with_dependants(objs, generic | sunk) + byid = {o["id"]: o for o in objs} # keep only top-surface support + objs = _drop_with_dependants(objs, [ + o["id"] for o in objs if o.get("parent") and ( + (p := byid.get(o["parent"])) is None or not support_ok(o, p))]) + nofront = {o["id"] for o in objs if o.get("front_known") is False} + objs = _drop_with_dependants(objs, nofront) # shown as fixed below (with what stands on them dropped) + if any(min(o["size"]) < 0 for o in objs): # only what is placed must have a size (0 = flat, shown as 1 cm) + return None, "bad_numbers" + kept = {o["id"] for o in objs} + children = {o["id"] for o in allobj if o.get("parent")} + # every other box that stands on the floor (or is a wall-hung unit, anchor "fixed") is told as fixed + for o in allobj: + nc = norm_cat(o["category"]) + if o["id"] in kept or o["id"] in children or o.get("parent") or o["size"][2] < FLAT_Z: + continue + if not (-FURNITURE_SUNK_Z <= o["pos"][2] <= FLOOR_Z or o["anchor"] == "fixed" or o["id"] in nofront): + continue + if hanging(o["category"]) or WALL_ART.search(nc) or head(o) in SOFT_COVERS: + continue + kind = "generic" if o["id"] in generic else "no_front" if o["id"] in nofront else \ + "floating" if o["anchor"] == "fixed" else "wall" + fixed.append({**{k: v for k, v in o.items() if k not in ("anchor", "parent", "tilted", "anchor_inferred")}, + "fixed_kind": kind}) + shell = Polygon(room["boundary"]).buffer(0) + fixed = _keep_fixed(fixed, objs, shell) + shown = kept | {f["id"] for f in fixed} + # a dropped box that stood on the floor and is neither shown nor an opening / flush finish leaves an unexplained + # hole in the reference layout; counted: its part on this room's floor that no placed object covers + inner = shell.buffer(-WALL_BAND_M) + # what explains a dropped box: a placed object or a fixed box standing on it (a rug does not) + covered = unary_union([Polygon(footprint(o)) for o in objs if not o.get("parent") and o["size"][2] >= FLAT_Z] + + [Polygon(footprint(f)) for f in fixed if f["size"][2] >= FLAT_Z]) + polygon = room.get("boundary_type") == "polygon" # (a rug does not explain a hole in the layout) + low = lambda o: o["pos"][2] <= FLOOR_Z and o["pos"][2] + o["size"][2] >= FLAT_Z # stands on / sticks out of it + hidden = sum(Polygon(footprint(o)).intersection(shell).difference(covered).area for o in room["objects"] + if o["id"] not in shown and o["id"] not in children and not o.get("parent") and low(o) + and not hanging(o["category"]) + and not WALL_ART.search(norm_cat(o["category"])) and not _flush(o, inner, polygon)) + if len(objs) < a.min_objects: + return None, "object_count" + # snap to the DISPLAYED numbers (cm, integer degrees) so checks, constraints and eval see what the model sees; + # z: floor 0, child = parent top; fixed boxes keep their own z (a window sill), floor-level ones go to 0 + rounded = lambda o: {**o, "size": shown_size(o), "yaw": math.radians(round(math.degrees(o["yaw"])) % 360)} + objs = [{**rounded(o), **({"z_src": o["pos"][2]} if not o.get("parent") and abs(o["pos"][2]) > 1e-9 else {})} + for o in objs] # provenance: the source z of a floor box snapped to 0 + fixed = [{**rounded(o), "pos": [num(o["pos"][0]), num(o["pos"][1]), + 0 if -FLOOR_Z <= o["pos"][2] <= FLOOR_Z else num(o["pos"][2])]} for o in fixed] + byid = {o["id"]: o for o in objs} + done = set() + + def z_of(o): + if o["id"] not in done: + p = byid.get(o.get("parent")) + o["pos"] = [num(o["pos"][0]), num(o["pos"][1]), num(z_of(p) + p["size"][2]) if p else 0] + done.add(o["id"]) + return o["pos"][2] + for o in objs: + z_of(o) + b = clean_boundary(room["boundary"]) + if b is None or (a.max_vertices and len(b) > a.max_vertices) or Polygon(b).area < MIN_AREA_M2: + return None, "boundary_shape" + out = {**room, "boundary": b, "objects": objs, "fixed": fixed} + # quality FLAGS (the room is kept; every flag is counted in stats.json and written next to the sample so a + # training sampler can weight or exclude it; --reject_flagged turns chosen flags back into rejections) + flags = {} + area = Polygon(b).area + geometry = check(out, oob_tol=a.oob_tol) + oob = geometry["oob"] + if oob: # reference objects crossing the walls by more than oob_tol + flags["oob_objects"] = len(oob) + if hidden > 0: # unexplained floor area of dropped boxes (see above) + flags["hidden_m2"] = round(hidden, 2) + if hidden >= a.hidden_max: + flags["hidden_obstacle"] = True + if area < 1.0: + flags["small_area_m2"] = round(area, 2) + if any(not o.get("parent") and head(o) not in FLAT and o["size"][2] >= 0.5 + and o["size"][0] * o["size"][1] >= 0.8 * area for o in objs): # composite built-in boxing the room + flags["room_filling_object"] = True + if _bad_overlap(objs): # copies of one wardrobe through each other + flags["overlapping_furniture"] = True + through = geometry["fixed_blocking"] # reuse the same box intersections for fixed obstacles + if through: + flags["fixed_collision"] = len(through) + if len(objs) == 1: + flags["single_object"] = True + snapped = sum(abs(o.get("z_src") or 0) > FLOOR_Z for o in objs) # beyond ordinary floor noise + if snapped: + flags["z_snapped"] = snapped + if any(o.get("tilted") for o in objs): + flags["tilted"] = sum(bool(o.get("tilted")) for o in objs) + if nofront: + flags["no_front_fixed"] = len(nofront) + hit = [k for k in a.reject_flagged if k in flags] + if hit: + return None, "flagged:" + hit[0] + out["meta"] = {**out.get("meta", {}), "flags": flags} + h = out.get("height") + if h and any(o["pos"][2] + o["size"][2] > h + 0.05 for o in objs): # height contradicts the boxes: unknown + out = {**out, "height": None, "meta": {**out["meta"], "height_dropped": True}} + elif h: + out = {**out, "height": num(h)} # the value the model is shown + return out, None + + +def _bad_overlap(objs): + """Copies of the same furniture that pass through each other (3D box overlap over 30% of the smaller footprint): + a wardrobe or bookcase entered twice in the source. Different categories are not judged here: at box level an + appliance built into a cabinet run, a loft bed over its desk or an L-sofa's box over its table look the same as a + real interpenetration, so those stay collisions reported by evaluate against the GT rate.""" + fl = [o for o in objs if not o.get("parent") and head(o) not in FLAT | SOFT_HEADS + and not COMPOSITE.search(norm_cat(o["category"]))] + fps = [Polygon(footprint(o)) for o in fl] + for i, a in enumerate(fl): + for j in range(i + 1, len(fl)): + b = fl[j] + if norm_cat(a["category"]) != norm_cat(b["category"]): + continue + if min(a["pos"][2] + a["size"][2], b["pos"][2] + b["size"][2]) - max(a["pos"][2], b["pos"][2]) <= 0.05: + continue + small, big = sorted((fps[i].area, fps[j].area)) + if fps[i].intersection(fps[j]).area > 0.3 * small and small >= big / 3: + return True + return False + + +def exact_key(room): + """Same boundary and the same objects and fixed boxes (category, size, pose) up to a 90-degree turn.""" + keys = [] + for k in range(4): + r = rot90(room, k) + pose = lambda o, c: (c, tuple(o["size"]), num(o["pos"][0]), num(o["pos"][1]), num(o["pos"][2]), + round(math.degrees(o["yaw"])) % 360) + keys.append(repr((r["boundary"], sorted(pose(o, norm_cat(o["category"])) for o in r["objects"]), + sorted(pose(o, slug(o["category"])) for o in r.get("fixed") or [])))) + return hashlib.sha1(min(keys).encode()).hexdigest() + + +def dedup(rooms): + """Same group in several sources (e.g. a ScanNet scan in Scan2CAD and InternScenes): keep the source whose + rooms of that group keep more objects; ties go to a polygon boundary, then source name.""" + by = collections.defaultdict(lambda: collections.defaultdict(list)) + for r in rooms: + by[r["group"]][r["source"]].append(r) + keep, dropped = [], collections.Counter() + for g, srcs in by.items(): + best = max(srcs, key=lambda s: (sum(len(r["objects"]) for r in srcs[s]), + any(r["boundary_type"] == "polygon" for r in srcs[s]), s)) + keep += srcs[best] + for s, rs in srcs.items(): + if s != best: + dropped[s] += len(rs) + return keep, dropped + + +def rng_for(uid, salt): + return random.Random(int(hashlib.sha1(f"{salt}:{uid}".encode()).hexdigest(), 16)) + + +def extract_constraints(room, rng, max_k=4): + """1..max_k relations that hold on the reference layout, one per type first (round robin, rarest type first). + Only a few relations are given, never the full relation graph: the model must still lay out everything else.""" + b = room["boundary"] + objs = room["objects"] + fps = {o["id"]: Polygon(footprint(o)) for o in objs} + cand = collections.defaultdict(list) + objs = [o for o in objs if head(o) not in FLAT and o["size"][2] >= 0.05] # rugs/flat boxes: not meaningful arguments + idx = {o["id"]: o for o in objs} # built once, not in every holds() call + for o in objs: + if o.get("parent"): + cand["on"].append(["on", o["id"], o["parent"]]) + if room["boundary_type"] == "polygon" and not o.get("parent") and holds(["against_wall", o["id"]], objs, b, idx): + cand["against_wall"].append(["against_wall", o["id"]]) + for x in objs: + for y in objs: + if x is y or x.get("parent") or y.get("parent"): + continue + if head(x) in FRONT and not fps[x["id"]].intersects(fps[y["id"]]) and \ + holds(["faces", x["id"], y["id"]], objs, b, idx): + cand["faces"].append(["faces", x["id"], y["id"]]) + if x["id"] < y["id"]: + gap = fps[x["id"]].distance(fps[y["id"]]) + d = next((d for d in (0.1, 0.2, 0.3, 0.5) if d >= gap), None) + small = min(fps[x["id"]].area, fps[y["id"]].area) + if small > 0 and fps[x["id"]].intersection(fps[y["id"]]).area > 0.3 * small: + continue # a collision or a duplicate annotation, not a "near" relation + if d is not None and holds(["near", x["id"], y["id"], d], objs, b, idx): # the one checker decides + cand["near"].append(["near", x["id"], y["id"], d]) + floor = [o for o in objs if not o.get("parent")] + cat = [norm_cat(o["category"]) for o in floor] + # only footprints the p-q segment can touch are tested (STRtree bbox query, then the same checks in floor order): + # the same candidates as testing every m, without the O(n^3) sweep (a 965-book library room took hours) + # the checks of holds(["between", m, p, q]) (segment p-q crosses m's footprint, not only touching it) and of the + # trivial-endpoint skip, evaluated vectorized on those candidates; same results, same floor order + geo = np.array([fps[m["id"]] for m in floor], dtype=object) + tree = STRtree(list(geo)) + for i, p in enumerate(floor): + for j in range(i + 1, len(floor)): + if cat[i] != cat[j]: + continue + q = floor[j] + seg = LineString([p["pos"][:2], q["pos"][:2]]) + ks = np.sort(tree.query(seg)) + if not len(ks): + continue + g = geo[ks] + ok = ~shapely.intersects(g, Point(p["pos"][:2])) & ~shapely.intersects(g, Point(q["pos"][:2])) \ + & shapely.intersects(g, seg) & ~shapely.touches(g, seg) & (ks != i) & (ks != j) + for k in ks[ok]: + cand["between"].append(["between", floor[k]["id"], p["id"], q["id"]]) + types = [t for t in PRIORITY if cand[t]] # rarest types first, so all five are covered + for t in types: + rng.shuffle(cand[t]) + k, out = rng.randint(1, max_k), [] + while len(out) < k and any(cand[t] for t in types): + for t in types: + if cand[t] and len(out) < k: + out.append(cand[t].pop()) + return out + + +def count_constraints(st, split, room, cons): + """stats: constraints per type; 'on' split by evidence (support given by the source vs box-contact inference).""" + objs = {o["id"]: o for o in room["objects"]} + for c in cons or []: + t = c[0] + if t == "on": + t += "(inferred)" if objs[c[1]].get("anchor_inferred") else "(source)" + st[f"{split}_constraints:{t}"] += 1 + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--ir", required=True, help="dir of .jsonl from fastfill.adapters") + ap.add_argument("--out", required=True, help="new dataset directory (must be absent or empty)") + ap.add_argument("--sources", nargs="*", default=None, help="default: every file in --ir") + ap.add_argument("--min_objects", type=int, default=1, help="rooms with fewer placeable objects have no target") + ap.add_argument("--max_vertices", type=int, default=0, help="reject rooms with more boundary vertices (0: no limit)") + ap.add_argument("--reject_flagged", nargs="*", default=[], + help="quality flags that reject a room instead of being recorded (oob_objects hidden_obstacle " + "small_area_m2 room_filling_object overlapping_furniture single_object)") + ap.add_argument("--keep_duplicates", action="store_true", help="keep exact duplicate layouts inside a split") + ap.add_argument("--boundary_types", nargs="*", default=["polygon", "hull"]) + ap.add_argument("--anchors", nargs="*", default=["floor", "object"], help="object anchors kept (see anchors.py)") + ap.add_argument("--oob_tol", type=float, default=0.10, help="flag oob_objects: reference objects outside boundary+tol") + ap.add_argument("--hidden_max", type=float, default=0.3, help="flag hidden_obstacle: m2 of unexplained dropped floor boxes") + ap.add_argument("--cap", nargs="*", default=[], help="SOURCE=N: keep N train rooms of SOURCE (hash-sampled); " + "default none: balance sources at sampling time, not here") + ap.add_argument("--source_anchors", nargs="*", default=[], help="SOURCE=a,b: anchors kept for that source") + ap.add_argument("--dev", type=float, default=0.05) + ap.add_argument("--test", type=float, default=0.05) + ap.add_argument("--constraint_frac", type=float, default=0.0, help="share of train rooms given as S2 (v1.1: 0.3)") + ap.add_argument("--no_rot90", action="store_true") + ap.add_argument("--field_dropout", type=float, default=0.2, help="train: drop room_type / height each with p") + ap.add_argument("--desc_dropout", type=float, default=0.5, help="train: drop all desc of a room with p") + a = ap.parse_args() + a.source_anchors = {k: v.split(",") for k, v in (x.split("=") for x in a.source_anchors)} + caps = {k: int(v) for k, v in (x.split("=") for x in a.cap)} + + files = _input_files(a, ap) + _validate_destination(a.out) + # Hash only implementation modules, excluding local audit/test artifacts. + here = Path(__file__).resolve().parent + code_files = _code_files(here) + code_sha = {str(p.relative_to(here)): _sha256(p) for p in code_files} + ir_sha = {p.name: _sha256(p) for p in files} + rooms, stats = [], collections.defaultdict(collections.Counter) + for f in files: + for line in _json_lines(f): + r0 = line + st = stats[r0["source"]] + r, why = prep(r0, a) + st["scanned"] += 1 + if why: + st["rejected:" + why] += 1 + continue + rooms.append({**r, "meta": {**r["meta"], "n_src_objects": len(r0["objects"])}}) + print(f"prepared {f.name}: {len(rooms)} retained rooms so far", flush=True) + if not rooms: + ap.error("no usable rooms; output was not changed") + rooms, dropped = dedup(rooms) + for s, n in dropped.items(): + stats[s]["dedup_dropped"] += n + + splits = assign_splits(rooms, a.dev, a.test) + # exact duplicates (same boundary, and the same objects and fixed boxes at the same poses, up to a 90-degree + # turn) stay one room per split: 3D-FRONT/SpatialLM repeat whole designs up to 26 times + first = {} + for i in sorted(range(len(rooms)), key=lambda i: hashlib.sha1(rooms[i]["uid"].encode()).hexdigest()): + key = (splits[i], exact_key(rooms[i])) + if key in first and not a.keep_duplicates: + stats[rooms[i]["source"]]["duplicate_layout"] += 1 + splits[i] = "duplicate" + else: + first[key] = i + for src, n in caps.items(): # hash order, not file order (files are sorted by type) + idx = sorted((i for i, r in enumerate(rooms) if r["source"] == src and splits[i] == "train"), + key=lambda i: hashlib.sha1(rooms[i]["uid"].encode()).hexdigest()) + for i in idx[n:]: + splits[i] = "capped" + if not any(s in ("train", "dev", "test") for s in splits): + ap.error("no rooms remain after filtering; output was not changed") + destination = Path(a.out).absolute() + destination.parent.mkdir(parents=True, exist_ok=True) + with tempfile.TemporaryDirectory(prefix=f".{destination.name}-", dir=destination.parent) as staging: + _write_dataset(rooms, splits, stats, a, Path(staging), code_sha, ir_sha) + if any(_sha256(p) != ir_sha[p.name] for p in files): + raise RuntimeError("input IR changed during build; output was not published") + dependencies = {"__init__.py", "build.py", "anchors.py", "scene.py", "split.py", "validate.py"} + current_code = {str(p.relative_to(here)): _sha256(p) for p in _code_files(here)} + if any(current_code.get(name) != code_sha.get(name) for name in dependencies): + raise RuntimeError("implementation changed during build; output was not published") + # Serving/training tools do not participate in the build. Record their release-time versions, + # retaining the start snapshot so concurrent unrelated edits are not attributed to data generation. + manifest_path = Path(staging) / "MANIFEST.json" + with manifest_path.open(encoding="utf-8") as f: + manifest = json.load(f) + with manifest_path.open("w", encoding="utf-8") as f: + json.dump({**manifest, "code_sha256": current_code, "build_start_code_sha256": code_sha}, + f, indent=1, allow_nan=False) + _validate_destination(destination) + os.rename(staging, destination) + for k, v in sorted(stats.items()): + print(f"{k:24s} {dict(v)}") + + +def _write_dataset(rooms, splits, stats, a, directory, code_sha, ir_sha): + with ExitStack() as stack: + open_output = lambda name: stack.enter_context(open(directory / f"{name}.jsonl", "w", encoding="utf-8")) + outs = {s: open_output(s) for s in ("train", "dev", "test")} + ir_outs = {s: open_output(s) for s in + ("dev_rooms", "test_rooms", "dev_constrained_rooms", "test_constrained_rooms")} + for r, s in zip(rooms, splits): + st = stats[r["source"]] + if s in ("capped", "duplicate"): + st["capped"] += s == "capped" + continue + # composition of what is WRITTEN (after dedup, duplicates and caps) + st["height_dropped"] += bool(r.get("meta", {}).get("height_dropped")) + st["objects_dropped"] += r["meta"]["n_src_objects"] - len(r["objects"]) - len(r["fixed"]) + st["fixed_items"] += len(r["fixed"]) + for f in r["fixed"]: + st["fixed:" + f["fixed_kind"]] += 1 + st["rooms_with_fixed"] += bool(r["fixed"]) + st["objects_max"] = max(st["objects_max"], len(r["objects"])) + for k in r["meta"]["flags"]: + st["flag:" + k] += 1 + st["supports_inferred"] += sum(bool(o.get("anchor_inferred")) for o in r["objects"] if o.get("parent")) + st["supports_evidence"] += sum(not o.get("anchor_inferred") for o in r["objects"] if o.get("parent")) + if s == "train": + # one random stream per purpose: --constraint_frac changes nothing but the constraints (clean ablation: + # v1.1 minus its constraints is v1.0 byte for byte) + aug, s2, drop = (rng_for(r["uid"], k) for k in ("aug", "s2", "dropout")) + c = canonical(r if a.no_rot90 else rot90(r, aug.randrange(4))) + cons = extract_constraints(c, s2) if s2.random() < a.constraint_frac else None + st["train_S2" if cons else "train_S1"] += 1 + count_constraints(st, "train", c, cons) + if drop.random() < a.field_dropout: + c = {**c, "room_type": None} + if drop.random() < a.field_dropout: + c = {**c, "height": None} + m = messages(c, cons, with_desc=drop.random() >= a.desc_dropout) + else: + c = canonical(r) + st[s] += 1 + m = messages(c) + ir_outs[f"{s}_rooms"].write(json.dumps(c, ensure_ascii=False, allow_nan=False) + "\n") + cons = extract_constraints(c, rng_for(r["uid"], "eval")) + if cons: + count_constraints(st, s, c, cons) + ir_outs[f"{s}_constrained_rooms"].write(json.dumps({**c, "constraints": cons}, ensure_ascii=False, allow_nan=False) + "\n") + outs[s].write(json.dumps({"uid": r["uid"], "source": r["source"], "flags": r["meta"]["flags"], "messages": m}, + ensure_ascii=False, allow_nan=False) + "\n") + with open(directory / "stats.json", "w", encoding="utf-8") as f: + json.dump({k: dict(v) for k, v in sorted(stats.items())}, f, indent=1, allow_nan=False) + manifest = {"args": vars(a), + "files": {p.name: {"sha256": _sha256(p), "lines": _line_count(p)} + for p in sorted(directory.iterdir()) if p.suffix == ".jsonl" or p.name == "stats.json"}, + "code_sha256": code_sha, "ir_sha256": ir_sha} + with open(directory / "MANIFEST.json", "w", encoding="utf-8") as f: + json.dump(manifest, f, indent=1, allow_nan=False) + + +def _sha256(path): + digest = hashlib.sha256() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _line_count(path): + with open(path, "rb") as f: + return sum(1 for _ in f) + + +def _code_files(here): + return sorted(p for p in here.rglob("*.py") + if not any(part.startswith(("review", ".", "__pycache__")) or part == "tests" + for part in p.relative_to(here).parts[:-1])) + + +def _json_lines(path): + with open(path, encoding="utf-8") as f: + for line_number, line in enumerate(f, 1): + try: + yield json.loads(line) + except json.JSONDecodeError as exc: + raise ValueError(f"{path}:{line_number}: invalid JSON: {exc.msg}") from exc + + +def _input_files(a, ap): + if not Path(a.ir).is_dir(): + ap.error(f"IR directory does not exist: {a.ir}") + files = sorted(Path(a.ir).glob("*.jsonl")) + if a.sources is not None: + missing = set(a.sources) - {p.stem for p in files} + if missing: + ap.error("missing IR sources: " + ", ".join(sorted(missing))) + files = [p for p in files if p.stem in a.sources] + if not files or any(p.stat().st_size == 0 for p in files): + ap.error("selected IR files must exist and be nonempty") + if any(not 0 <= x <= 1 for x in (a.dev, a.test, a.constraint_frac, a.field_dropout, a.desc_dropout)) or \ + a.dev + a.test > 1: + ap.error("split/dropout/constraint fractions must be in [0, 1], with dev + test <= 1") + return files + + +def _validate_destination(path): + path = Path(path) + if path.is_symlink() or (path.exists() and (not path.is_dir() or any(path.iterdir()))): + raise FileExistsError(f"refusing to replace existing dataset: {path}; choose a new version directory") + + +if __name__ == "__main__": + main() diff --git a/code/fastfill/download.py b/code/fastfill/download.py new file mode 100644 index 0000000000000000000000000000000000000000..e0b93e7ef0ab51ede97aae2305838a6de38c30df --- /dev/null +++ b/code/fastfill/download.py @@ -0,0 +1,116 @@ +"""Download the exported 3D room collections. + +Default = aggregate files only (files at most 1 level below each export folder: the merged +json/jsonl exports, README, kits). --full adds per-scene copies and raw sources (~90k small files). +Files are fetched one by one with per-file backoff and skipped when already present with the right +size, so an interrupted run resumes; the Hub rate-limits bursts of small files (HTTP 429/499). + +Usage: + python -m fastfill.download --out /Volumes/harddisk/3D_Room_Collections + python -m fastfill.download --out DIR --only IL3D 3D-FRONT --full [--with-meshes] +""" +import argparse +import fnmatch +import os +import time +from concurrent.futures import ThreadPoolExecutor + +from huggingface_hub import HfApi, hf_hub_download +from huggingface_hub.hf_api import RepoFile + +# name -> (repo_id, folder inside the repo or "" for the whole repo); from excel/3D_Room_Scene_Collection.xlsx +SOURCES = { + "SAGE-10k": ("liantian/3D_Room_Scene_Collections", "SAGE10k"), + "M3DLayout": ("imChuling/3D_Room_Collections", "M3DLayout_exported"), + "InteriorGS": ("imChuling/3D_Room_Collections", "InteriorGS_exported"), + "MetaScenes": ("imChuling/3D_Room_Collections", "MetaScenes_exported"), + "IL3D": ("imChuling/3D_Room_Collections", "IL3D_exported"), + "SceneSmith": ("imChuling/3D_Room_Collections", "SceneSmith_exported"), + "MansionWorld": ("imChuling/3D_Room_Collections", "MansionWorld_exported"), + "OptiScene": ("imChuling/3D_Room_Collections", "OptiScene_exported"), + "InternScenes": ("imChuling/3D_Room_Collections", "InternScenes_exported"), + "Structured3D": ("imChuling/3D_Room_Collections", "Structured3D_exported"), + "3RScan": ("imChuling/3D_Room_Collections", "3RScan_exported"), + "ARKitScenes": ("imChuling/3D_Room_Collections", "ARKitScenes_exported"), + "ScanNet": ("imChuling/3D_Room_Collections", "ScanNet_exported"), + "SceneCAD_Scan2CAD": ("imChuling/3D_Room_Collections", "SceneCAD_Scan2CAD_exported"), + "SpatialLM": ("imChuling/3D_Room_Collections", "SpatialLM_exported"), + "3D-FRONT": ("BillLin66/3D_Room_Collections", "3DFront_exported"), + "ScanNet++": ("BillLin66/3D_Room_Collections", "ScanNet++_exported"), + "SpatialGen": ("BillLin66/3D_Room_Collections", "SpatialGen_exported"), + "ASE": ("BillLin66/3D_Room_Collections", "ASE_exported"), + "HM3DSem": ("BillLin66/3D_Room_Collections", "HM3DSem_exported"), + "HSSD-200": ("XXXpilar/hssd_clean", ""), + "MultiScan": ("XXXpilar/multiscan-clean", ""), +} + +# meshes, textures, images and per-vertex segment archives: not needed for text layout training +HEAVY = ["*.glb", "*.png", "*.jpg", "*.jpeg", "*.gif", "*.obj", "*.ply", "*.npz", "*.tar", "*.mtl", "*.urdf", + "*segments_anno.zip", "*keyframes.zip"] + + +def select(files, folder, full, with_meshes): + """Pick repo paths for one export folder.""" + pre = f"{folder}/" if folder else "" + # whole-repo sources (HSSD, MultiScan) keep their layout json one level deeper: scenes//scene.json + max_depth = 2 if folder == "" else 1 + out = [] + for p in files: + if not p.startswith(pre): + continue + rel = p[len(pre):] + if not with_meshes and any(fnmatch.fnmatch(p, pat) for pat in HEAVY): + continue + if full or rel.count("/") <= max_depth: + out.append(p) + return out + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--out", required=True) + ap.add_argument("--only", nargs="*", default=None, help="subset of SOURCES names") + ap.add_argument("--full", action="store_true", help="also per-scene/raw files deeper in the folder") + ap.add_argument("--with-meshes", action="store_true") + ap.add_argument("--workers", type=int, default=8) + a = ap.parse_args() + by_repo = {} + for name in a.only or SOURCES: + repo, folder = SOURCES[name] + by_repo.setdefault(repo, []).append((name, folder)) + api = HfApi() + failed_count = 0 + for repo, items in by_repo.items(): + # one listing per repo: listing a large repo takes minutes + sizes = {f.path: f.size for f in api.list_repo_tree(repo, repo_type="dataset", recursive=True) + if isinstance(f, RepoFile)} + local = f"{a.out}/{repo.replace('/', '__')}" + for name, folder in items: + wanted = select(sizes, folder, a.full, a.with_meshes) + todo = [p for p in wanted if not (os.path.isfile(f"{local}/{p}") and os.path.getsize(f"{local}/{p}") == sizes[p])] + print(f"== {name}: {repo}/{folder} ({len(wanted)} files, {len(todo)} to fetch, " + f"{sum(sizes[p] for p in todo) / 1e9:.2f} GB)", flush=True) + with ThreadPoolExecutor(a.workers) as ex: + failed = [p for p, ok in zip(todo, ex.map(lambda p: fetch(repo, p, local), todo)) if not ok] + failed_count += len(failed) + print(f"== {name}: done" if not failed else f"== {name}: {len(failed)} files failed, rerun to resume", + flush=True) + if failed_count: + raise SystemExit(1) + + +def fetch(repo, path, local, tries=8): + """One file with backoff: the Hub answers bursts of small-file requests with 429/499/5xx.""" + for k in range(tries): + try: + hf_hub_download(repo, path, repo_type="dataset", local_dir=local) + return True + except Exception as e: # noqa: BLE001 network/HTTP errors of several libraries + if k == tries - 1: + print(f" failed {path}: {str(e)[:120]}", flush=True) + return False + time.sleep(min(30 * 2 ** k, 600)) + + +if __name__ == "__main__": + main() diff --git a/code/fastfill/download_full.sh b/code/fastfill/download_full.sh new file mode 100644 index 0000000000000000000000000000000000000000..c0a2e3d8ef42d72faf16a3ec6cb0ee113b4cfdc4 --- /dev/null +++ b/code/fastfill/download_full.sh @@ -0,0 +1,27 @@ +#!/bin/zsh +# Full download of the exported 3D room collections with `hf download`, one dataset after another (needs zsh). +# fastfill/download_full.sh OUT_DIR [NAME ...] (names as in fastfill/download.py SOURCES; default: all 22) +# Re-runnable: files already in OUT_DIR are skipped. A dataset is retried until `hf download` exits cleanly, +# because the Hub answers folders with tens of thousands of small files with transient 429/499 errors. +# Exits 1 on an unknown name or when a dataset still fails after 30 attempts. +OUT=${1:?usage: download_full.sh OUT_DIR [NAME ...]}; shift +OUT=${OUT:A} # absolute before the cd below +HF=${HF_BIN:-hf}; PY=${PYTHON:-python} +cd "${0:A:h}/.." || exit 1 +known=($($PY -c 'from fastfill.download import SOURCES; print(*SOURCES)')) || exit 1 +for n in "$@"; do (( ${known[(Ie)$n]} )) || { echo "unknown dataset: $n (known: $known)"; exit 1; }; done +failed=() +$PY -c 'from fastfill.download import SOURCES +for n, (r, f) in SOURCES.items(): print(n, r, f or "-")' | while read name repo folder; do + if (( $# )) && [[ ${@[(Ie)$name]} -eq 0 ]]; then continue; fi + inc=(); [[ $folder != "-" ]] && inc=(--include "$folder/*") + echo "== $name: $repo/$folder" + ok=0 + for i in {1..30}; do + $HF download $repo --repo-type dataset $inc --local-dir "$OUT/${repo/\//__}" --max-workers 4 >/dev/null && { ok=1; break; } + echo "== $name: attempt $i failed, retrying in 60s"; sleep 60 + done + if (( ok )); then echo "== $name: done"; else echo "== $name: FAILED after 30 attempts"; failed+=($name); fi +done +(( ${#failed} )) && { echo "== FAILED: $failed"; exit 1; } +echo "== FULL DOWNLOAD FINISHED" diff --git a/code/fastfill/evaluate.py b/code/fastfill/evaluate.py new file mode 100644 index 0000000000000000000000000000000000000000..bbd40f43c79f9485fd8fdc39fc93fe201485b512 --- /dev/null +++ b/code/fastfill/evaluate.py @@ -0,0 +1,230 @@ +"""Generate layouts for held-out rooms and score them. + + python -m fastfill.evaluate --model outputs/ff-v1-merged --rooms data/fastfill_v1/test_rooms.jsonl --out eval/ff-v1 + python -m fastfill.evaluate --gt-only --rooms data/fastfill_v1/test_rooms.jsonl --out eval/gt # reference layouts + python -m fastfill.evaluate # self-check + +Rooms come from fastfill.build (already canonical: ids and order as the model saw them in training). +Every request is counted: unparsable or incomplete outputs, and prompts of --max_len tokens or more (prompt_too_long, +never generated), are invalid and fail all their constraints (no retries, no filtering); identical requests are scored +once. Metrics are reported per source and pooled separately for +in-distribution sources and held-out (eval-only) sources. +Validity, 1 mm containment, acceptance, per-object and collision rates and OOR are reported next to their GT value +(`*_gt`): the reference layout of the same rooms scored exactly like the model's answer (repair and serve.py's +acceptance included), over the same rows (all requests, or the complete answers for the per-object and collision +rates). Constraints hold on the reference by construction, and it always parses. +Uses vLLM when installed, else HF generate (up to --batch rows of equal token length). Greedy decoding; each answer may use the context left after its prompt +(--max_len, the model's sequence limit), so nothing is truncated below what training saw. +""" +import argparse +import collections +import copy +import itertools +import json +import os +import sys + +from shapely.geometry import Polygon + +from fastfill.interface import snap_inside +from fastfill.scene import apply, footprint, gt_placements, messages, parse, prompt_text +from fastfill.validate import check, holds + +BUCKETS = ("main", "severe", "tuck", "contained", "fixed") + + +def generate(model, prompts, max_len, batch): + """One text per prompt; None for a prompt of max_len tokens or more (nothing fits after it and vLLM raises on it), + which score() records as prompt_too_long.""" + res = [None] * len(prompts) + try: + from vllm import LLM, SamplingParams + llm = LLM(model=model, max_model_len=max_len) + tok = llm.get_tokenizer() + fit = [(i, max_len - len(tok(p)["input_ids"])) for i, p in enumerate(prompts)] + fit = [(i, n) for i, n in fit if n > 0] + outs = llm.generate([prompts[i] for i, _ in fit], [SamplingParams(temperature=0, max_tokens=n) for _, n in fit]) + for (i, _), o in zip(fit, outs): + res[i] = o.outputs[0].text + except ImportError: + import torch + from transformers import AutoModelForCausalLM, AutoTokenizer + tok = AutoTokenizer.from_pretrained(model, padding_side="left") + m = AutoModelForCausalLM.from_pretrained(model, dtype=torch.bfloat16, device_map="auto") + lengths = sorted((len(tok(p, add_special_tokens=False)["input_ids"]), i) for i, p in enumerate(prompts)) + # A shared generation limit is safe only for equal-length prompts; padding otherwise steals short rows' budget. + for length, group in itertools.groupby(lengths, key=lambda row: row[0]): + if length >= max_len: + continue + fit = [i for _, i in group] + for k in range(0, len(fit), batch): + idx = fit[k:k + batch] + enc = tok([prompts[i] for i in idx], return_tensors="pt", padding=True, add_special_tokens=False).to(m.device) + out = m.generate(**enc, max_new_tokens=max_len - length, do_sample=False) + for i, t in zip(idx, tok.batch_decode(out[:, enc["input_ids"].shape[1]:], skip_special_tokens=True)): + res[i] = t + return res + + +def geometry(room): + c = check(room) + return {"valid": c["valid"], "oob_objects": len(c["oob"]), "support_fail": len(c["support_fail"]), + "ceiling_objects": len(c["ceiling"]), "oor": c["oor"], "coll_fixed": len(c["fixed_collisions"]), + **{f"coll_{k}": len(c["collisions"][k]) for k in BUCKETS[:4]}} + + +def score(room, text, error=None): + """`text` None scores the reference layout (--gt-only, qa_report); `error` (prompt_too_long) marks a request the + model never answered, counted as a failed request like an unparsable answer. r["gt"] = judge() of the reference.""" + ids = [o["id"] for o in room["objects"]] + gt = judge(room, gt_placements(room)) + r = {"uid": room["uid"], "source": room["source"], "group": room["group"], "n": len(ids), + "held_out": bool(room.get("meta", {}).get("eval_only")), "gt": gt} + pl, err = ({}, error) if error else parse(text, ids) if text is not None else (None, None) + r["error"] = err + r["parsed"] = not error and err not in ("truncated_or_bad_json", "bad_json") + cons = room.get("constraints") or [] + if err is not None: # a failed request fails every constraint it was asked to satisfy + return {**r, "valid": False, "constraints": [[c[0], False] for c in cons], + "constraints_repaired": [[c[0], False] for c in cons]} + return {**r, **(gt if text is None else judge(room, pl))} + + +def judge(room, pl): + """One complete answer (parse() form) scored raw, and after the interface's <= 10 cm push back inside (reported + apart: the raw proposal is the model's result).""" + cons = room.get("constraints") or [] + pred = apply(room, pl) + r = geometry(pred) + r["constraints"] = [[c[0], holds(c, pred["objects"], room["boundary"])] for c in cons] + floor = Polygon(room["boundary"]) + rep = copy.deepcopy(pl) + snap_inside(rep, room, floor, cons, room["boundary"]) # never breaks a constraint that held (all are hard) + fixed = apply(room, rep) + c = check(fixed) + r["inside_1mm"], r["inside_1mm_repaired"] = inside(pred, floor), inside(fixed, floor) + r["valid_repaired"] = c["valid"] + r["constraints_repaired"] = [[k[0], holds(k, fixed["objects"], room["boundary"])] for k in cons] + r["repaired_objects"] = sum(rep[i]["pos"] != pl[i]["pos"] for i in pl) + # what serve.py accepts: valid after repair, every object within 1 mm of the walls, nothing through a non-window + # fixed box, every constraint holding + r["accepted"] = c["valid"] and r["inside_1mm_repaired"] and not c["fixed_blocking"] \ + and all(v for _, v in r["constraints_repaired"]) + return r + + +def inside(room, floor, tol=1e-3): + """Every object, on others too, inside the boundary within 1 mm (EmbodiedGen's containment rule).""" + grown = floor.buffer(tol) + return all(grown.covers(Polygon(footprint(o))) for o in room["objects"]) + + +def summarize(rows): + def agg(rs): + n = len(rs) + if not n: + return None + ok = [r for r in rs if r["error"] is None] + objs = sum(r["n"] for r in ok) + s = {"rooms": n, "groups": len({r["group"] for r in rs}), + "parse_rate": sum(r["parsed"] for r in rs) / n, "complete_rate": len(ok) / n, + "errors": dict(collections.Counter(r["error"] for r in rs if r["error"]))} + # over every request (a failed one counts as failing); *_repaired after the interface's deterministic repair + # (not model skill); accepted = serve.py's 200, repair included; GT: the reference layouts of the same rooms + for k, key in (("valid_rate", "valid"), ("valid_rate_repaired", "valid_repaired"), + ("inside_1mm_rate", "inside_1mm"), ("inside_1mm_rate_repaired", "inside_1mm_repaired"), + ("accepted_rate", "accepted")): + s[k] = sum(r.get(key, False) for r in rs) / n + s[f"{k}_gt"] = sum(r["gt"][key] for r in rs) / n + # geometry is measured on complete answers only; with none it is unknown (None), not a perfect 0 + rate = lambda num, den: num / den if den else None + for k, key in (("repaired_object_rate", "repaired_objects"), ("oob_object_rate", "oob_objects"), + ("support_fail_rate", "support_fail"), ("ceiling_object_rate", "ceiling_objects")): + s[k] = rate(sum(r[key] for r in ok), objs) + s[f"{k}_gt"] = rate(sum(r["gt"][key] for r in ok), objs) + for k in BUCKETS: # share of rooms with a collision of this kind: model vs GT on the same (complete) rooms + name = "fixed_collision" if k == "fixed" else k + s[f"rooms_with_{name}"] = rate(sum(r[f"coll_{k}"] > 0 for r in ok), len(ok)) + s[f"rooms_with_{name}_gt"] = rate(sum(r["gt"][f"coll_{k}"] > 0 for r in ok), len(ok)) + s["mean_oor"] = rate(sum(r["oor"] for r in ok), len(ok)) + s["mean_oor_gt"] = rate(sum(r["gt"]["oor"] for r in ok), len(ok)) + by_c, by_r = collections.defaultdict(list), collections.defaultdict(list) + for r in rs: # every request, failed ones included + for t, v in r.get("constraints", []): + by_c[t].append(v) + for t, v in r.get("constraints_repaired", []): + by_r[t].append(v) + s["constraint_satisfaction"] = {t: sum(v) / len(v) for t, v in sorted(by_c.items())} + s["constraint_satisfaction_repaired"] = {t: sum(v) / len(v) for t, v in sorted(by_r.items())} + return s + by = collections.defaultdict(list) + for r in rows: + by[r["source"]].append(r) + # eval-only sources (SpatialGen, SceneSmith) are out of distribution: never pooled with the rest + return {"in_dist": agg([r for r in rows if not r["held_out"]]), "held_out": agg([r for r in rows if r["held_out"]]), + **{k: agg(v) for k, v in sorted(by.items())}} + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--rooms", required=True, help="dev_rooms.jsonl / test_rooms.jsonl from fastfill.build") + ap.add_argument("--out", required=True) + ap.add_argument("--model", help="merged checkpoint dir or HF id") + ap.add_argument("--gt-only", action="store_true", help="score the reference layouts themselves") + ap.add_argument("--max_len", type=int, default=40960, help="model sequence limit; an answer may use what its prompt leaves") + ap.add_argument("--batch", type=int, default=1, help="HF generate only: prompts per forward pass (at 40,960 context " + "the KV cache of more than one row does not fit on one GPU)") + ap.add_argument("--limit", type=int, default=None) + a = ap.parse_args() + + uniq = {} # identical requests (same room text and constraints) are scored once + for line in open(a.rooms): + r = json.loads(line) + uniq.setdefault(messages(r, r.get("constraints"), with_target=False)[1]["content"], r) + rooms = list(uniq.values())[:a.limit] + if a.gt_only: + texts = [None] * len(rooms) + else: + from transformers import AutoTokenizer + tok = AutoTokenizer.from_pretrained(a.model) + prompts = [prompt_text(tok, messages(r, r.get("constraints"), with_target=False)) for r in rooms] + texts = generate(a.model, prompts, a.max_len, a.batch) + # a None text after generation is a prompt the model could not take (>= --max_len tokens): a failed request + rows = [score(r, t, "prompt_too_long" if t is None and not a.gt_only else None) for r, t in zip(rooms, texts)] + os.makedirs(a.out, exist_ok=True) + with open(f"{a.out}/predictions.jsonl", "w") as f: + for r, t in zip(rows, texts): + f.write(json.dumps({**r, "output": t}, ensure_ascii=False) + "\n") + summary = summarize(rows) + json.dump(summary, open(f"{a.out}/metrics.json", "w"), indent=1) + print(json.dumps({k: summary[k] for k in ("in_dist", "held_out")}, indent=1)) + + +if __name__ == "__main__": + if sys.argv[1:]: + main() + else: + from fastfill.scene import target_json + # the reference chair stands 3 cm through the south wall facing a box 28.5 deg off its front: not inside 1 mm, + # repaired by (-4, +3) cm (the shortest push would break the faces constraint), then accepted + room = {"uid": "u", "source": "S", "group": "g", "boundary": [[0, 0], [4, 0], [4, 3], [0, 3]], + "boundary_type": "polygon", "height": 2.8, "fixed": [], "constraints": [["faces", "chair_1", "box_1"]], + "objects": [{"id": "chair_1", "category": "chair", "size": [.5, .5, .9], "pos": [1, .22, 0], "yaw": 0.0}, + {"id": "box_1", "category": "box", "size": [.06, .06, .3], "pos": [1.35, .03, 0], "yaw": 0.0}]} + gt = score(room, None) + assert not gt["inside_1mm"] and gt["inside_1mm_repaired"] and gt["repaired_objects"] == 1 and gt["accepted"], gt + assert all(gt[k] == v for k, v in gt["gt"].items()) and score(room, target_json(room))["accepted"] + # a model answer 20 cm through the west wall (oob, not repaired, faces lost) and a prompt that did not fit + off = target_json({**room, "objects": [{**room["objects"][0], "pos": [.05, 1.5, 0]}, room["objects"][1]]}) + rows = [gt, score(room, off), score(room, None, "prompt_too_long")] + assert rows[1]["oob_objects"] == 1 and rows[1]["gt"] == gt["gt"] and rows[2]["gt"] == gt["gt"] + s = summarize(rows)["in_dist"] + want = {"valid_rate": (1 / 3, 1), "accepted_rate": (1 / 3, 1), "inside_1mm_rate": (0, 0), + "inside_1mm_rate_repaired": (1 / 3, 1), "valid_rate_repaired": (1 / 3, 1), + "oob_object_rate": (1 / 4, 0), "repaired_object_rate": (1 / 4, 2 / 4), "support_fail_rate": (0, 0)} + assert all((s[k], s[f"{k}_gt"]) == v for k, v in want.items()), s + assert s["constraint_satisfaction"] == {"faces": 1 / 3} and s["rooms_with_main_gt"] == 0, s + # --gt-only: every rate equals its GT + s = summarize([gt])["in_dist"] + assert all(s[k] == s[f"{k}_gt"] for k in s if f"{k}_gt" in s), s + print("evaluate.py self-check ok") diff --git a/code/fastfill/inspect_ir.py b/code/fastfill/inspect_ir.py new file mode 100644 index 0000000000000000000000000000000000000000..1693519248a29c5b22346cb91be8f8c49ff97301 --- /dev/null +++ b/code/fastfill/inspect_ir.py @@ -0,0 +1,87 @@ +"""Diagnostics for IR files: counts, GT geometry checks, and the front-direction test. + + python -m fastfill.inspect_ir /Volumes/harddisk/fastfill_ir/IL3D.jsonl [--limit 5000] + +Front test: for wall-hugging categories (bed, sofa, wardrobe, toilet, ...) the side of the footprint that +touches the boundary should be the back, i.e. local -X under the IR convention (front = local +X). +A different dominant side means the adapter's front_offset_deg is wrong. +""" +import argparse +import collections +import json +import math +import re + +from shapely.geometry import Point, Polygon + +from fastfill.validate import check + +WALL_HUGGERS = re.compile(r"\b(bed|sofa|couch|wardrobe|closet|toilet|dresser|bookcase|bookshelf|tv[ _]?stand|" + r"refrigerator|fridge|sideboard|nightstand|armoire)\b", re.I) + + +def wall_side(o, ring, tol=0.15): + """Which local axis side (+X, -X, +Y, -Y) of the footprint lies on the boundary, or None.""" + c, s = math.cos(o["yaw"]), math.sin(o["yaw"]) + hx, hy = o["size"][0] / 2, o["size"][1] / 2 + x, y = o["pos"][0], o["pos"][1] + best, name = tol, None + for lab, (dx, dy) in {"+X": (hx, 0), "-X": (-hx, 0), "+Y": (0, hy), "-Y": (0, -hy)}.items(): + d = ring.distance(Point(x + c * dx - s * dy, y + s * dx + c * dy)) + if d < best: + best, name = d, lab + return name + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("files", nargs="+") + ap.add_argument("--limit", type=int, default=None) + a = ap.parse_args() + for path in a.files: + n, nobj, btypes, rtypes, cats = 0, [], collections.Counter(), collections.Counter(), collections.Counter() + meta = collections.Counter() + geo = collections.Counter() + sides = collections.defaultdict(collections.Counter) + for line in open(path): + r = json.loads(line) + n += 1 + nobj.append(len(r["objects"])) + btypes[r.get("boundary_type")] += 1 + rtypes[r.get("room_type")] += 1 + meta["rooms_with_tilted"] += bool(r["meta"].get("n_tilted")) + meta["rooms_with_incomplete"] += bool(r["meta"].get("n_incomplete")) + for o in r["objects"]: + cats[o["category"]] += 1 + if r.get("boundary") and r["objects"] and (a.limit is None or geo["rooms"] < a.limit): + c = check(r) + geo["rooms"] += 1 + geo["objects"] += c["n"] + geo["valid"] += c["valid"] + geo["oob_objects"] += len(c["oob"]) + geo["rooms_with_collision"] += bool(c["collisions"]["main"]) + geo["oor_sum"] += c["oor"] + ring = Polygon(r["boundary"]).exterior + for o in r["objects"]: + m = WALL_HUGGERS.search(o["category"] or "") + if m: + sides[m.group(1).lower()][wall_side(o, ring)] += 1 + nobj.sort() + q = lambda f: nobj[min(int(f * len(nobj)), len(nobj) - 1)] if nobj else None + print(f"== {path}\nrooms {n} objects/room p10 {q(.1)} p50 {q(.5)} p90 {q(.9)} max {q(1)}") + print("boundary_type", dict(btypes), "\nmeta", dict(meta)) + print("room_type top", rtypes.most_common(12)) + print("category top", cats.most_common(25), f"({len(cats)} distinct)") + if geo["rooms"]: + g = geo + print(f"GT geometry on {g['rooms']} rooms: valid {g['valid'] / g['rooms']:.3f} " + f"oob_object_rate {g['oob_objects'] / max(g['objects'], 1):.3f} " + f"rooms_with_main_collision {g['rooms_with_collision'] / g['rooms']:.3f} mean_oor {g['oor_sum'] / g['rooms']:.3f}") + print("front test (side touching the wall; expect -X):") + for k, v in sorted(sides.items(), key=lambda kv: -sum(kv[1].values()))[:10]: + tot = sum(v.values()) + print(f" {k:12s} n={tot:6d} " + " ".join(f"{s}:{v[s] / tot:.2f}" for s in ("-X", "+X", "-Y", "+Y", None))) + + +if __name__ == "__main__": + main() diff --git a/code/fastfill/interface.py b/code/fastfill/interface.py new file mode 100644 index 0000000000000000000000000000000000000000..3b333f67129589b7d8fd58beeb27cb0a0ca74111 --- /dev/null +++ b/code/fastfill/interface.py @@ -0,0 +1,641 @@ +"""WorldEdge request (scope doc, Appendix B) <-> FastFill model text. + + room, ctx = request_to_room(request) + prompt = prompt_text(tok, messages(room, room["constraints"], with_target=False)) + ... one generation call ... + result = placements_from_text(text, room, ctx) + +Model side (scene.py): room frame with the boundary AABB min corner at (0, 0), object local +X = semantic front, +integer-degree yaw inside the text. Interface side: the request's own frame (boundary_xy, floor_z), position_m of +the box bottom center, yaw_rad about +Z of the object's own local frame, support_parent = floor_id or the id of the +object it stands on, support_surface = "top". Objects anchored to walls or ceilings ("anchor": "wall" | "ceiling") +and fronts that are not along a horizontal local axis are returned as unsupported instead of being sent to the model, +and so is any constraint that names one of them. +Floor/object anchors are enforced after generation and reported in validation.constraints as ["anchor", id, kind], +with hard=true, trained=false, model_conditioned=false: the existing model protocol cannot express them. +room.fixed_geometry (optional): boxes already in the room that nothing may be placed on or through (doors, windows, +columns, stairs, furniture that stays), each {"id", "category"?, "size_xyz_m", "position_m" (bottom center), +"yaw_rad"?, "fixed_kind"?}; shown to the model as `fixed`, never placed, and overlaps with them are reported in validation +(fixed_collisions), and any such overlap with a non-window box fails the layout (fixed_blocking, serve.py 422). Normalized as build.prep shaped training: |z| <= FLOOR_Z is snapped to +0; a box starting at or above FIXED_MAX_Z, covering half the room or more, or standing clear of the walls is not shown +and is reported in unsupported as "fixed_ignored". The wall band is 0.30 m for fixed_kind="structure", otherwise +0.10 m; if fixed_kind is omitted, anchors.is_structure(category) selects the band. An explicit kind takes precedence. +A constraint may not name a fixed box (v1 relates objects to place). + +layout_constraints (Appendix B names). Relational constraints are shown to the model as must-hold; after repair a failed +"hard" one (the default; "hard": false marks a soft one, which is only reported) fails the layout: + {"type": "faces", "subject", "target"} {"type": "between", "subject", "anchors": [a, b]} + {"type": "supported_by", "subject", "parent", "surface": "top"} {"type": "against_wall", "subject"} + {"type": "near", "subject", "target", "max_gap_m"} footprint gap, NOT centre distance (trained on 0.1-0.5) + {"type": "keepout", "polygon_xy": [[x, y], ...]} hard: cut out of the floor, padded by KEEPOUT_PAD +Soft keepouts leave the floor unchanged. They are reported as ["keepout", polygon_xy] in request coordinates, +with hard=false, trained=false, model_conditioned=false; the existing model protocol has no advisory region field. +A soft keepout holds when no placed footprint overlaps it by more than 1e-9 square meters (touching is allowed). +against_wall is accepted against the real walls, not keepout cuts, although the model sees a cut as wall. +A padded keepout that stops less than WALL_SNAP short of a wall (or of another keepout) is snapped to it: the strip +left between them is dropped from the floor the model sees (cm rounding folded one under 5 mm into a self-intersection, +a bad_boundary 422, for 35 of 20,000 EmbodiedGen-shaped requests); floor cut off behind such a gap is a pocket, and a +gap that alone would cut off POCKET_M2 or more is left open. A cut that still folds under cm rounding (a gap under 5 mm +to a large corner, a keepout crossing a wall by under 5 mm) is keepout_not_representable, not bad_boundary. +Forms training never contains (build.extract_constraints) are still shown and checked as asked, but reported with +"trained": false in validation: against_wall in a hull room, faces whose subject's category head is not in +build.FRONT, against_wall / near / faces / between naming an object that stands on another, between with anchors of +two categories, an argument (of supported_by: the subject) that is a FLAT covering or under 5 cm tall. +Rejected with ValueError (HTTP 422 in serve.py): malformed or non-finite numbers, a coordinate_system other than +m / right / Z / bbox_bottom_center / rad, duplicate ids, unknown anchors, non-positive sizes or height, constraints +naming unknown objects, fixed boxes or themselves, 'on' cycles or two parents, a keepout that splits the room or leaves an +island (enclosed pockets under POCKET_M2 are treated as unusable floor). +""" +import copy +import math + +from shapely.affinity import translate +from shapely.errors import ShapelyError +from shapely.geometry import Point, Polygon +from shapely.ops import nearest_points, unary_union + +from fastfill.anchors import FIXED_MAX_Z, FLAT, FLOOR_Z, is_structure +from fastfill.build import FRONT, THICK_WALL_M, WALL_BAND_M + +FLAT_Z = 0.15 # = build.FLAT_Z: a sunk fixed box whose top stays under this is not shown (checked in __main__) +from fastfill.scene import apply, canonical, clean_boundary, footprint, head, norm_cat, num, parse +from fastfill.validate import check, holds, support_contains + +HULL_SOURCES = {"convex_hull", "hull", "scan_hull", "floor_hull"} +KEEPOUT_PAD = 0.08 # < validate.check's oob tolerance (0.10) minus cm rounding: repair onto the keepout edge stays valid +POCKET_M2 = 0.5 # free floor enclosed by keepouts and walls smaller than this is dropped, not a reason to refuse +WALL_SNAP = 0.01 # a strip of floor narrower than this left by a padded keepout (at a wall or keepout) is dropped +ANCHORS = {None, "floor", "object", "wall", "ceiling"} +COORDS = {"units": "m", "handedness": "right", "up_axis": "Z", "position_reference": "bbox_bottom_center", + "yaw_units": "rad"} +SNAP_TOL = 0.10 # training layouts cross walls by up to this much (40% of test rooms by > 1 mm); EmbodiedGen allows 1 mm +SNAP_GRID = sorted(((dx / 100, dy / 100) for dx in range(-10, 11) for dy in range(-10, 11) if dx * dx + dy * dy <= 100), + key=lambda d: d[0] ** 2 + d[1] ** 2) + + +def request_to_room(req): + """-> (canonical model room with constraints, ctx for mapping the answer back). Raises ValueError on bad input.""" + try: + return _request_to_room(req) + except (KeyError, TypeError, IndexError, AttributeError, OverflowError, ShapelyError) as e: + raise ValueError(f"bad_request: {e!r}") from e + + +def _finite(v, lo=None, above=None): + if not (isinstance(v, (int, float)) and not isinstance(v, bool) and math.isfinite(v) and abs(v) < 1e4 + and (lo is None or v >= lo) and (above is None or v > above)): + raise ValueError(f"bad_number: {v!r}") + return float(v) + + +def _text(v, optional=True): + if not (isinstance(v, str) or (optional and v is None)): + raise ValueError(f"bad_text: {v!r}") + return v + + +def _request_to_room(req): + rm = req["room"] + for k, v in (req.get("coordinate_system") or {}).items(): + if k in COORDS and v != COORDS[k]: + raise ValueError(f"unsupported coordinate_system {k}={v!r} (FastFill speaks {COORDS})") + cons_in = req.get("layout_constraints") or [] + pts = rm["boundary_xy"] + keepout_constraints = [c for c in cons_in if c["type"] == "keepout"] + for c in keepout_constraints: + if not isinstance(c.get("hard", True), bool): + raise ValueError(f"malformed constraint: {c}") + for p in [q for poly in [pts, *(c["polygon_xy"] for c in keepout_constraints)] for q in poly]: + _finite(p[0]), _finite(p[1]) + for c in keepout_constraints: + poly = Polygon(c["polygon_xy"]) + if not poly.is_valid or poly.is_empty or poly.area <= 0: + raise ValueError("bad_keepout") + keepouts = [c["polygon_xy"] for c in keepout_constraints if c.get("hard", True)] + if rm.get("height_m") is not None: + _finite(rm["height_m"], above=0) + _text(rm.get("room_type")) + walls = Polygon(pts) # against_wall is checked against these, not against keepout cuts + floor = walls # what answers are snapped into: the walls minus the keepouts themselves + if keepouts: # cut out, padded, from what the model sees: it treats a keepout as wall + if not floor.is_valid: + raise ValueError("bad_boundary") + cut = unary_union([Polygon(k) for k in keepouts]) + seen = floor.difference(cut.buffer(KEEPOUT_PAD, join_style="mitre")) + # opening by WALL_SNAP / 2: strips under WALL_SNAP wide vanish, everything wider comes back edge for edge; the + # intersection clips the mitre corners that re-grow past a slanted wall where a strip vanished. A gap whose + # closing alone would cut off POCKET_M2 or more stays open (9 of 20,000 EmbodiedGen-shaped requests: two + # padded keepouts 5-10 mm apart in front of a corner): the corner behind it stays floor, as without the snap + snap = seen.intersection(seen.buffer(-WALL_SNAP / 2, join_style="mitre").buffer(WALL_SNAP / 2, join_style="mitre")) + big = lambda g: sum(q.area >= POCKET_M2 for q in getattr(g, "geoms", [g])) + seen = seen if big(snap) > 1 and big(seen) == 1 else snap + parts = sorted(getattr(seen, "geoms", [seen]), key=lambda g: -g.area) + pockets = [g for g in parts[1:] if g.area < POCKET_M2] + if seen.is_empty or len(parts) - len(pockets) != 1 or parts[0].interiors: + raise ValueError("keepout_not_representable") # fills or splits the room, or leaves an island + seen = parts[0] + floor = floor.difference(unary_union([cut, *pockets])) + pts = seen.exterior.coords[:-1] + x0, y0 = (min(p[i] for p in pts) for i in (0, 1)) + boundary = clean_boundary([[x - x0, y - y0] for x, y in pts]) + if boundary is None: # walls that round cleanly: the cut folds under cm rounding (a sub-5 mm gap or wall mouth) + wx, wy = walls.bounds[:2] + ok = keepouts and clean_boundary([[x - wx, y - wy] for x, y in walls.exterior.coords[:-1]]) + raise ValueError("keepout_not_representable" if ok else "bad_boundary") + ctx = {"x0": x0, "y0": y0, "floor_z": _finite(rm.get("floor_z", 0.0)), + "floor_id": _text(rm.get("floor_id", "floor"), optional=False), + "floor": translate(floor, -x0, -y0), "walls": [[x - x0, y - y0] for x, y in walls.exterior.coords[:-1]], + "soft_keepouts": [(copy.deepcopy(c["polygon_xy"]), translate(Polygon(c["polygon_xy"]), -x0, -y0)) + for c in keepout_constraints if not c.get("hard", True)], + "front_k": {}, "order": [], "unsupported": []} + fixed, fixed_ids = [], set() + near, wall = walls.buffer(WALL_BAND_M), walls.buffer(THICK_WALL_M) + for f in rm.get("fixed_geometry") or []: + _text(f["id"], optional=False), _text(f.get("category")) + kind = _text(f.get("fixed_kind")) + structural = kind == "structure" if kind is not None else is_structure(f.get("category")) + sx, sy, sz = (_finite(v, above=0) for v in f["size_xyz_m"]) + x, y, z = (_finite(v) for v in f["position_m"]) + yaw = _finite(f.get("yaw_rad", 0.0)) % (2 * math.pi) + fixed_ids.add(f["id"]) + z -= ctx["floor_z"] + fp = Polygon(footprint({"pos": [x, y], "size": [sx, sy], "yaw": yaw})) + # as build.prep / anchors.fixed_geometry shaped training: nothing starting at FIXED_MAX_Z or above, no slab + # covering half the room, nothing clear of the walls; a floor-level bottom is z=0 + if z + sz < FLAT_Z: # a covering or trim (or wholly under the floor): not shown, as in build._keep_fixed + ctx["unsupported"].append({"id": f["id"], "reason": "fixed_ignored"}) + continue + if z < -FLOOR_Z: # sunk below the floor: cut at the floor + sz, z = z + sz, 0.0 + if z >= FIXED_MAX_Z or fp.area >= 0.5 * walls.area or not fp.intersects(wall if structural else near): + ctx["unsupported"].append({"id": f["id"], "reason": "fixed_ignored"}) + continue + fixed.append({"id": f["id"], "ext_id": f["id"], "category": f.get("category") or "structure", + "size": [sx, sy, sz], "pos": [x - x0, y - y0, 0.0 if abs(z) <= FLOOR_Z else z], "yaw": yaw}) + ids = [o["id"] for o in req["objects_to_place"]] + [f["id"] for f in rm.get("fixed_geometry") or []] + if len(set(ids)) < len(ids) or ctx["floor_id"] in ids: + raise ValueError("duplicate_object_id") + objs = [] + for o in req["objects_to_place"]: + _text(o["id"], optional=False), _text(o["category"], optional=False), _text(o.get("desc")) + sx, sy, sz = (_finite(v, above=0) for v in o["size_xyz_m"]) + if o.get("anchor") not in ANCHORS: + raise ValueError(f"unknown anchor: {o.get('anchor')!r}") + fx, fy, fz = (_finite(v) for v in (o.get("semantic_front_local") or [1, 0, 0])) + phi, h = math.atan2(fy, fx), math.hypot(fx, fy) + k = round(phi / (math.pi / 2)) % 4 + if o.get("anchor") in ("wall", "ceiling"): + ctx["unsupported"].append({"id": o["id"], "reason": "unsupported_anchor"}) + continue + if h < 1e-9 or abs(fz) > 1e-3 * h or abs(math.remainder(phi - k * math.pi / 2, 2 * math.pi)) > 1e-3: + ctx["unsupported"].append({"id": o["id"], "reason": "front_not_axis_aligned"}) + continue + if k % 2: # front along local Y: the model's local X is that axis + sx, sy = sy, sx + ctx["front_k"][o["id"]] = k + ctx["order"].append(o["id"]) + objs.append({"id": o["id"], "ext_id": o["id"], "category": o["category"], "size": [sx, sy, sz], + "desc": o.get("desc"), "anchor": o.get("anchor")}) + room = canonical({"room_type": rm.get("room_type"), "boundary": boundary, "height": rm.get("height_m"), + "boundary_type": "hull" if str(rm.get("boundary_source")).lower() in HULL_SOURCES else "polygon", + "fixed": fixed, "objects": objs}) + m = {o["ext_id"]: o["id"] for o in room["objects"]} + skipped = {u["id"] for u in ctx["unsupported"]} + cons, ctx["checks"] = [], [] # what the model is shown / what the answer is accepted against + for c in cons_in: + t = c["type"] + if t == "keepout": + continue + s = c["subject"] + anchors = c.get("anchors", []) + if not isinstance(anchors, list) or not isinstance(c.get("hard", True), bool): + raise ValueError(f"malformed constraint: {c}") + refs = [s] + [c[k] for k in ("target", "parent") if k in c] + anchors + if any(r in fixed_ids for r in refs): + raise ValueError(f"constraint names a fixed box (v1 constraints only relate objects to place): {c}") + if any(r not in m and r not in skipped for r in refs): + raise ValueError(f"constraint on an unknown object: {c}") + if len(set(refs)) < len(refs): + raise ValueError(f"constraint names the same object twice: {c}") + if any(r in skipped for r in refs): # the rest of the room is still placed + ctx["unsupported"].append({"constraint": c, "reason": "references_unsupported_object"}) + continue + if t == "faces": + k = ["faces", m[s], m[c["target"]]] + elif t == "between" and len(c["anchors"]) == 2: + k = ["between", m[s]] + [m[a] for a in c["anchors"]] + elif t == "supported_by" and c.get("surface", "top") == "top": + k = ["on", m[s], m[c["parent"]]] + elif t == "near": + k = ["near", m[s], m[c["target"]], _finite(c["max_gap_m"], lo=0)] + elif t == "against_wall": + k = ["against_wall", m[s]] + else: + raise ValueError(f"unsupported constraint: {c}") + # the model sees cm (near rounded DOWN, so meeting what it sees meets the request); acceptance uses the + # request's own number; hard=false constraints are reported but do not fail the layout + cons.append(k[:3] + [num(math.floor(k[3] * 100 + 1e-9) / 100)] if t == "near" else k) + ctx["checks"].append((k, c.get("hard", True))) + parent = {} + for k in cons: + if k[0] == "on": + if parent.setdefault(k[1], k[2]) != k[2]: + raise ValueError(f"{k[1]} is asked to stand on two objects") + for x in parent: # an 'on' cycle can never hold + seen, cur = {x}, parent[x] + while cur in parent: + if cur in seen: + raise ValueError(f"'on' cycle through {cur}") + seen.add(cur) + cur = parent[cur] + room["constraints"] = cons + return room, ctx + + +def snap_inside(pl, room, floor, cons=(), walls=None): + """Move each object that crosses `floor` (model frame) by at most SNAP_TOL back inside, with what stands on it. + Lower objects first, so a child is judged after its parent moved; a child is only moved as far as its centre + stays over its parent (validate.check's support rule), else it is left for validation to reject. A move is never + taken if it would deepen the overlap of the object OR of anything standing on it (the whole support subtree + moves with it) with a fixed box of the room (room["fixed"], at that object's height). + The hard constraints `cons` (validate.holds, against_wall against `walls`) that held before the repair hold after + it: if the plain pass breaks one, the repair is redone refusing every move that breaks one of them. The plain pass + goes first because a constraint may fail only between two moves (three chairs pushed off one wall in turn: the + between through the middle one fails until it moves too). Without this, pushing chairs off walls broke 7 GT-true + constraints in 8,254 test_constrained rooms (faces 4, against_wall 2, near 1).""" + size = {o["id"]: o["size"] for o in room["objects"]} + lay = lambda q: {j: {"pos": p["pos"], "yaw": p["yaw"], "size": size[j], "parent": p["on"]} for j, p in q.items()} + held = [c for c in cons if holds(c, None, walls, lay(pl))] + raw = copy.deepcopy(pl) + _push(pl, room, floor, size, [], walls) + if not all(holds(c, None, walls, lay(pl)) for c in held): + pl.update(raw) + _push(pl, room, floor, size, held, walls) + + +def _push(pl, room, floor, size, held, walls): + inside = floor.buffer(1e-3) # validation()'s tolerance + fixed = room.get("fixed") or [] + kids = {} + for i, p in pl.items(): + if p["on"]: + kids.setdefault(p["on"], []).append(i) + for i in sorted(pl, key=lambda i: pl[i]["pos"][2]): + p = pl[i] + box = lambda dx, dy: Polygon(footprint({"pos": [p["pos"][0] + dx, p["pos"][1] + dy], "size": size[i], "yaw": p["yaw"]})) + if inside.covers(box(0, 0)): + continue + q = pl.get(p["on"]) + seat = {**q, "size": size[p["on"]]} if q else None + push = [0.0, 0.0] # per axis, the largest move any outside corner needs to reach the boundary + for c in box(0, 0).exterior.coords[:-1]: + if not floor.covers(Point(c)): + q = nearest_points(floor.boundary, Point(c))[0] + for k, d in enumerate((q.x - c[0], q.y - c[1])): + if abs(d) > abs(push[k]): + push[k] = d + # the object and everything standing on it move together (parse() rejects support cycles); fixed boxes at + # each one's height (validate.check's rule): no move may deepen any of their overlaps + tree, todo = [], [i] + while todo: + j = todo.pop() + tree.append(j) + todo += kids.get(j, []) + at = lambda j, dx, dy: Polygon(footprint({"pos": [pl[j]["pos"][0] + dx, pl[j]["pos"][1] + dy], + "size": size[j], "yaw": pl[j]["yaw"]})) + blocks = {j: unary_union([Polygon(footprint(f)) for f in fixed if min( + pl[j]["pos"][2] + size[j][2], f["pos"][2] + f["size"][2]) - max(pl[j]["pos"][2], f["pos"][2]) > 0.05]) + for j in tree} + worst = {j: at(j, 0, 0).intersection(blocks[j]).area + 1e-6 for j in tree} + mine = [c for c in held if set(tree) & set(c[1:])] + + def keeps(dx, dy): # every held constraint on the subtree still holds after the move + ix = {j: {"pos": [q["pos"][0] + dx, q["pos"][1] + dy, q["pos"][2]] if j in tree else q["pos"], "yaw": q["yaw"], + "size": size[j], "parent": q["on"]} for j, q in pl.items()} if mine else {} + return all(holds(c, None, walls, ix) for c in mine) + # the nearest-boundary push, its axis parts, then any move within SNAP_TOL (1 cm grid, shortest first) + # (a nearest-boundary push over SNAP_TOL is dropped, the grid still tries every move within SNAP_TOL: a box + # stuck at a re-entrant corner may need a long diagonal push but a short move along one axis) + for dx, dy in [c for c in (tuple(push), (push[0], 0), (0, push[1])) if math.hypot(*c) <= SNAP_TOL] + SNAP_GRID: + if inside.covers(box(dx, dy)) and all(at(j, dx, dy).intersection(blocks[j]).area <= worst[j] for j in tree) \ + and (seat is None or support_contains(seat, [p["pos"][0] + dx, p["pos"][1] + dy])) and keeps(dx, dy): + for j in tree: + pl[j]["pos"] = [pl[j]["pos"][0] + dx, pl[j]["pos"][1] + dy, pl[j]["pos"][2]] + break + + +def placements_from_text(text, room, ctx): + """Model answer -> {"placements": after repair, "raw_placements": as the model proposed, "validation": + {"raw": ..., "repaired": ...}, "error": code | None, "unsupported": [...]}, placements in request order. + Repair = floor objects crossing a wall or a keepout by at most SNAP_TOL moved back inside (snap_inside); it is + deterministic post-processing, so the raw proposal and its validation are always returned next to it.""" + ids = [o["id"] for o in room["objects"]] + pl, err = parse(text, ids) + if err is not None: + return {"placements": [], "raw_placements": [], "validation": None, "error": err, + "unsupported": ctx["unsupported"]} + raw = copy.deepcopy(pl) + snap_inside(pl, room, ctx["floor"], [k for k, hard in ctx["checks"] if hard], ctx["walls"]) + return {"placements": _to_request(pl, room, ctx), "raw_placements": _to_request(raw, room, ctx), + "validation": {"raw": validation(raw, room, ctx), "repaired": validation(pl, room, ctx)}, + "error": None, "unsupported": ctx["unsupported"]} + + +def _to_request(pl, room, ctx): + model2ext = {o["id"]: o["ext_id"] for o in room["objects"]} + by_ext = {} + for mid, p in pl.items(): + ext = model2ext[mid] + yaw = (p["yaw"] - ctx["front_k"][ext] * math.pi / 2) % (2 * math.pi) # model +X (front) -> asset frame + item = {"id": ext, "position_m": [p["pos"][0] + ctx["x0"], p["pos"][1] + ctx["y0"], p["pos"][2] + ctx["floor_z"]], + "yaw_rad": yaw, "support_parent": model2ext[p["on"]] if p["on"] else ctx["floor_id"]} + if p["on"]: + item["support_surface"] = "top" + by_ext[ext] = item + return [by_ext[i] for i in ctx["order"]] + + +def trained(k, lay): + """False for a constraint form build.extract_constraints never makes, so the model never saw it (v2.1 train: 0 + against_wall in 5,117 constrained hull rooms; no faces with a non-FRONT subject; no against_wall / near / faces / + between with an argument on another object; no between with anchors of two categories; no argument (of 'on': the + subject) that is FLAT or under 5 cm tall). Reported, not refused: the caller decides. Thresholds (near 0.1-0.5) and + whether the relation held on a reference layout are not forms and are not checked here.""" + o = {x["id"]: x for x in lay["objects"]} + args = [o[i] for i in k[1:2 if k[0] == "on" else None] if i in o] + return not ((k[0] == "against_wall" and lay["boundary_type"] != "polygon") + or (k[0] == "faces" and head(o[k[1]]) not in FRONT) + or (k[0] != "on" and any(x.get("parent") for x in args)) + or any(head(x) in FLAT or x["size"][2] < 0.05 for x in args) + or (k[0] == "between" and norm_cat(o[k[2]]["category"]) != norm_cat(o[k[3]]["category"]))) + + +def validation(pl, room, ctx): + """The canonical checks (validate.check / validate.holds) on a placement, with request ids. ok = valid (inside + boundary + 0.1, supported, under the ceiling), every object (on others too) within 1 mm of the walls minus the + keepouts, and every hard constraint holding with the request's own numbers against the real walls. + An overlap with a fixed box other than a window (validate.check fixed_blocking: a column, a door, a stair) also + fails ok: those are physical obstacles; furniture in front of a window is reported in fixed_collisions only. + Collisions between placed objects are reported, not gated (a chair tucked under a table is a + box overlap; the reference layouts themselves have ~10% rooms with a severe one).""" + ext = {o["id"]: o["ext_id"] for o in room["objects"] + (room.get("fixed") or [])} + lay = apply(room, pl) + c = check(lay) + grown = ctx["floor"].buffer(1e-3) + outside = [ext[o["id"]] for o in lay["objects"] if not grown.covers(Polygon(footprint(o)))] + cons = [{"constraint": [k[0]] + [ext.get(v, v) for v in k[1:]], "hard": hard, + "holds": holds(k, lay["objects"], ctx["walls"]), "trained": trained(k, lay)} for k, hard in ctx["checks"]] + cons += [{"constraint": ["anchor", ext[o["id"]], o["anchor"]], "hard": True, + "holds": bool(o.get("parent")) == (o["anchor"] == "object"), + "trained": False, "model_conditioned": False} + for o in lay["objects"] if o.get("anchor") in ("floor", "object")] + cons += [{"constraint": ["keepout", pts], "hard": False, + "holds": all(Polygon(footprint(o)).intersection(zone).area <= 1e-9 for o in lay["objects"]), + "trained": False, "model_conditioned": False} for pts, zone in ctx["soft_keepouts"]] + return {"ok": c["valid"] and not outside and not c["fixed_blocking"] and all(x["holds"] for x in cons if x["hard"]), + "oob": [ext[i] for i in c["oob"]], + "support_fail": [ext[i] for i in c["support_fail"]], "ceiling": [ext[i] for i in c["ceiling"]], + "outside_1mm": outside, "collisions": {k: [[ext[a], ext[b]] for a, b in v] for k, v in c["collisions"].items()}, + "fixed_collisions": [[ext[a], ext[b]] for a, b in c["fixed_collisions"]], + "fixed_blocking": [[ext[a], ext[b]] for a, b in c["fixed_blocking"]], "constraints": cons} + + +if __name__ == "__main__": + import json + + from fastfill.scene import messages, target_json + from fastfill.validate import holds + + req = { # scope doc Appendix B, room moved to (10, 20) and one chair modelled with its front along local +Y + "room": {"id": "new_room", "room_type": "meeting room", "boundary_xy": [[10, 20], [14, 20], [14, 24], [10, 24]], + "boundary_source": "specified_polygon", "floor_id": "room_floor", "floor_z": 0.0, "height_m": 2.8}, + "objects_to_place": [ + {"id": "table_1", "category": "table", "size_xyz_m": [1.4, 0.8, 0.75]}, + {"id": "chair_1", "category": "chair", "size_xyz_m": [0.55, 0.5, 0.9], "semantic_front_local": [1, 0, 0]}, + {"id": "chair_2", "category": "chair", "size_xyz_m": [0.5, 0.55, 0.9], "semantic_front_local": [0, 1, 0]}, + {"id": "cup_1", "category": "cup", "size_xyz_m": [0.08, 0.08, 0.1]}, + {"id": "clock_1", "category": "clock", "size_xyz_m": [0.3, 0.05, 0.3], "anchor": "wall"}], + "layout_constraints": [ + {"type": "faces", "subject": "chair_1", "target": "chair_2", "hard": True}, + {"type": "faces", "subject": "chair_2", "target": "chair_1", "hard": True}, + {"type": "between", "subject": "table_1", "anchors": ["chair_1", "chair_2"], "hard": True}, + {"type": "supported_by", "subject": "cup_1", "parent": "table_1", "surface": "top", "hard": True}]} + room, ctx = request_to_room(req) + assert [o["id"] for o in room["objects"]] == ["table_1", "chair_1", "chair_2", "cup_1"], room["objects"] + assert room["objects"][2]["size"] == [0.55, 0.5, 0.9] # chair_2 re-framed: front axis first + assert ctx["unsupported"] == [{"id": "clock_1", "reason": "unsupported_anchor"}] + assert room["constraints"][3] == ["on", "cup_1", "table_1"] + # the model answers in its own frame: chair_1 at x=0.9 facing +X, chair_2 at x=3.1 facing -X, cup on the table + ans = {"table_1": ([2, 2, 0], 0), "chair_1": ([0.9, 2, 0], 0), "chair_2": ([3.1, 2, 0], math.pi), + "cup_1": ([1.7, 2, 0.75], 0)} + placed = {**room, "objects": [{**o, "pos": ans[o["id"]][0], "yaw": ans[o["id"]][1], + "parent": "table_1" if o["id"] == "cup_1" else None} for o in room["objects"]]} + assert all(holds(c, placed["objects"], placed["boundary"]) for c in room["constraints"]) + text = target_json(placed) + assert "constraints" in messages(room, room["constraints"], with_target=False)[1]["content"] + out = placements_from_text(text, room, ctx) + got = {p["id"]: p for p in out["placements"]} + assert out["error"] is None and [p["id"] for p in out["placements"]] == ["table_1", "chair_1", "chair_2", "cup_1"] + assert got["chair_1"]["position_m"] == [10.9, 22, 0] and got["chair_1"]["yaw_rad"] == 0 + assert abs(got["chair_2"]["yaw_rad"] - math.pi / 2) < 1e-9 # front on local +Y: asset turned by 90 deg + assert got["cup_1"]["support_parent"] == "table_1" and got["cup_1"]["support_surface"] == "top" + assert got["table_1"]["support_parent"] == "room_floor" and "support_surface" not in got["table_1"] + assert placements_from_text("not json", room, ctx)["error"] == "bad_json" + # table 3 cm through the east wall (x up to 4.03): moved back inside, the cup on it moves along; 20 cm is left alone + over = {**placed, "objects": [{**o, "pos": [3.33 if o["id"] == "table_1" else 3.0 if o["id"] == "cup_1" else o["pos"][0], + *o["pos"][1:]]} for o in placed["objects"]]} + got = {p["id"]: p["position_m"] for p in placements_from_text(target_json(over), room, ctx)["placements"]} + assert abs(got["table_1"][0] - 13.3) < 1e-9 and abs(got["cup_1"][0] - 12.97) < 1e-9, got + far = {**over, "objects": [{**o, "pos": [3.5, *o["pos"][1:]]} if o["id"] == "table_1" else o for o in over["objects"]]} + assert placements_from_text(target_json(far), room, ctx)["placements"][0]["position_m"][0] == 13.5 + + # a doorway keepout on the west wall becomes a notch (padded) in the floor the model sees; the frame is unchanged + door = {"type": "keepout", "polygon_xy": [[10, 21.5], [11, 21.5], [11, 22.5], [10, 22.5]]} + room2, ctx2 = request_to_room({**req, "layout_constraints": req["layout_constraints"] + [door]}) + assert (ctx2["x0"], ctx2["y0"]) == (10, 20) and [0, 1.42] in room2["boundary"] and [1.08, 2.58] in room2["boundary"] + assert room2["constraints"] == room["constraints"] + # a constraint naming an unsupported object is skipped and reported; the rest of the room is still placed + hang = {"type": "near", "subject": "clock_1", "target": "table_1", "max_gap_m": 0.5} + room3, ctx3 = request_to_room({**req, "layout_constraints": [hang]}) + assert room3["constraints"] == [] and ctx3["unsupported"][1] == {"constraint": hang, "reason": "references_unsupported_object"} + up = {**req, "objects_to_place": [{**req["objects_to_place"][0], "semantic_front_local": [0, 0, 1]}], "layout_constraints": []} + assert request_to_room(up)[1]["unsupported"] == [{"id": "table_1", "reason": "front_not_axis_aligned"}] + bad = [{**req, "objects_to_place": req["objects_to_place"] * 2}, # duplicate ids + {**req, "layout_constraints": [{"type": "near", "id": "cup_1", "target": "table_1", "max_distance_m": 1.5}]}, + {**req, "layout_constraints": [{"type": "between", "subject": "table_1", "anchors": ["chair_1"]}]}, + {**req, "room": {**req["room"], "floor_z": None}}, + {**req, "objects_to_place": [{**req["objects_to_place"][0], "size_xyz_m": [1.4, -0.8, 0.75]}]}, + {**req, "objects_to_place": [{**req["objects_to_place"][0], "size_xyz_m": [1.4, float("nan"), 0.75]}]}, + {**req, "layout_constraints": [{**door, "polygon_xy": [[9, 19], [15, 19], [15, 25], [9, 25]]}]}] # fills the room + obj0 = req["objects_to_place"][0] + bad += [{**req, "layout_constraints": [{"type": "supported_by", "subject": "cup_1", "parent": "cup_1"}]}, # self + {**req, "layout_constraints": [{"type": "supported_by", "subject": "cup_1", "parent": "table_1"}, + {"type": "supported_by", "subject": "table_1", "parent": "cup_1"}]}, # cycle + {**req, "layout_constraints": [{"type": "faces", "subject": "chair_1", "target": "chair_2", "hard": "false"}]}, + {**req, "layout_constraints": [{"type": "between", "subject": "table_1", "anchors": "cc"}]}, + {**req, "coordinate_system": {"units": "cm"}}, + {**req, "objects_to_place": [{**obj0, "anchor": "banana"}]}, + {**req, "objects_to_place": req["objects_to_place"] + [{"id": "x", "category": "clock", "size_xyz_m": [-1, 1, 1], + "anchor": "wall"}]}, + {**req, "room": {**req["room"], "room_type": 7}}, + {**req, "room": {**req["room"], "height_m": 0}}, + {**req, "objects_to_place": [{**obj0, "size_xyz_m": [1e200, 1, 1]}]}, + {**req, "room": {**req["room"], "floor_z": float("inf")}}, + {**req, "room": {**req["room"], "height_m": float("nan")}}, + {**req, "layout_constraints": [{"type": "near", "subject": "cup_1", "target": "table_1", "max_gap_m": -0.1}]}] + for b in bad: + try: + request_to_room(b) + raise AssertionError(b) + except ValueError: + pass + + # near: the model is shown the threshold rounded down to cm, the answer is accepted against the request's own + # number; a failed hard constraint fails the layout (after repair), a failed soft one is only reported + near = lambda gap, hard=True: {"type": "near", "subject": "chair_1", "target": "chair_2", "max_gap_m": gap, "hard": hard} + r5, c5 = request_to_room({**req, "layout_constraints": [near(0.126)]}) + assert r5["constraints"] == [["near", "chair_1", "chair_2", 0.12]] and c5["checks"][0][0][3] == 0.126 + apart = target_json(placed) # chairs 1.7 m apart: fails any near <= 0.5 + v = placements_from_text(apart, *request_to_room({**req, "layout_constraints": [near(0.1)]}))["validation"] + assert not v["repaired"]["ok"] and v["repaired"]["constraints"][0]["holds"] is False, v + v = placements_from_text(apart, *request_to_room({**req, "layout_constraints": [near(0.1, hard=False)]}))["validation"] + assert v["repaired"]["ok"], v + # a cup hanging 3 cm over the east wall from a table flush with it: the cup is pushed in, staying on the table + edge = {**placed, "objects": [{**o, "pos": [3.3, 2, 0] if o["id"] == "table_1" else [3.99, 2, 0.75] if o["id"] == "cup_1" + else o["pos"]} for o in placed["objects"]]} + out = placements_from_text(target_json(edge), room, ctx) + cup = {p["id"]: p for p in out["placements"]}["cup_1"]["position_m"] + assert out["validation"]["raw"]["outside_1mm"] == ["cup_1"] and out["validation"]["repaired"]["ok"], out["validation"] + assert abs(cup[0] - 13.96) < 1e-9, cup + # fixed geometry: a column at (12, 22) is shown to the model in the room frame, never placed, and a table on it is + # reported (not refused); a fixed id that repeats an object id is a duplicate + col = {"id": "col_1", "category": "column", "size_xyz_m": [0.3, 0.3, 2.8], "position_m": [12, 22, 0]} + room6, ctx6 = request_to_room({**req, "room": {**req["room"], "fixed_geometry": [col]}, "layout_constraints": []}) + assert room6["fixed"][0]["id"] == "column_1" and room6["fixed"][0]["pos"] == [2, 2, 0], room6["fixed"] + assert json.loads(messages(room6, [], with_target=False)[1]["content"])["fixed"] == [ + {"id": "column_1", "size": [0.3, 0.3, 2.8], "pos": [2, 2, 0], "yaw": 0}] + out6 = placements_from_text(target_json(placed), room6, ctx6) # table_1 centred on the column + assert not out6["validation"]["repaired"]["ok"] and out6["validation"]["repaired"]["fixed_collisions"] == [["table_1", "col_1"]] + # the repair moves a table with the cup on it; a move that would push the CUP into a fixed upper cabinet (above the + # table top, at the cup's height) is not taken, so the table stays 3 cm through the wall and the layout fails + cab = {"id": "cab_1", "category": "cabinet", "size_xyz_m": [0.4, 0.6, 1.0], "position_m": [12.75, 22, 0.78]} + room9, ctx9 = request_to_room({**req, "room": {**req["room"], "fixed_geometry": [cab]}, "layout_constraints": []}) + moved = {"table_1": [3.33, 2, 0], "cup_1": [3.0, 2, 0.75], "chair_2": [1.5, 3.5, 0]} + lay9 = {**placed, "objects": [{**o, "pos": moved.get(o["id"], o["pos"])} for o in placed["objects"]]} + out9 = placements_from_text(target_json(lay9), room9, ctx9) + got = {p["id"]: p["position_m"] for p in out9["placements"]} + assert abs(got["table_1"][0] - 13.33) < 1e-9 and abs(got["cup_1"][0] - 13.0) < 1e-9, got + assert not out9["validation"]["repaired"]["fixed_collisions"] and "table_1" in out9["validation"]["repaired"]["outside_1mm"] + for b in ({**col, "id": "table_1"}, {**col, "size_xyz_m": [0, 0.3, 2.8]}): # duplicate id; zero size + try: + request_to_room({**req, "room": {**req["room"], "fixed_geometry": [b]}}) + raise AssertionError(b) + except ValueError: + pass + # a constraint may not name a fixed box: refused with its own message + try: + request_to_room({**req, "room": {**req["room"], "fixed_geometry": [col]}, + "layout_constraints": [{"type": "near", "subject": "table_1", "target": "col_1", "max_gap_m": 0.5}]}) + raise AssertionError("constraint on a fixed box accepted") + except ValueError as e: + assert "fixed box" in str(e), e + # training-side normalization: |z| <= FLOOR_Z snapped to 0; a box starting at FIXED_MAX_Z or above, covering half + # the room, or standing clear of it is not shown and reported as fixed_ignored + fx = lambda **kw: {**col, **kw} + room8, ctx8 = request_to_room({**req, "layout_constraints": [], "room": {**req["room"], "floor_z": 1.0, "fixed_geometry": [ + fx(id="door", position_m=[12, 20.5, 1.06]), fx(id="beam", size_xyz_m=[4, 0.3, 0.3], position_m=[12, 22, 2.5]), + fx(id="slab", size_xyz_m=[3, 3, 0.1], position_m=[12, 22, 1]), fx(id="far", position_m=[30, 30, 1])]}}) + assert [f["ext_id"] for f in room8["fixed"]] == ["door"] and room8["fixed"][0]["pos"] == [2, 0.5, 0], room8["fixed"] + assert [u["id"] for u in ctx8["unsupported"] if u["reason"] == "fixed_ignored"] == ["beam", "slab", "far"], ctx8["unsupported"] + # repair never deepens the overlap with a fixed box: a table lying along the east wall 3 cm out (yaw 90, x 3.23..4.03) + # with its west edge 8 cm over a column at (3.3, 2) is left where it is (every move within SNAP_TOL pushes it further + # in, so it stays outside_1mm and serve.py refuses the layout); the same table clear of the column is moved back + col2 = {**col, "position_m": [13.3, 22, 0]} + room7, ctx7 = request_to_room({**req, "room": {**req["room"], "fixed_geometry": [col2]}, "layout_constraints": []}) + side = lambda y: {**placed, "objects": [{**o, "pos": [3.63, y, o["pos"][2]], "yaw": math.pi / 2 if o["id"] == "table_1" else o["yaw"]} + if o["id"] in ("table_1", "cup_1") else o for o in placed["objects"]]} + out7 = placements_from_text(target_json(side(2.0)), room7, ctx7) + got = {p["id"]: p["position_m"] for p in out7["placements"]} + assert abs(got["table_1"][0] - 13.63) < 1e-9 and out7["validation"]["repaired"]["outside_1mm"] == ["table_1"], out7["validation"] + assert ["table_1", "col_1"] in out7["validation"]["repaired"]["fixed_collisions"] + got = {p["id"]: p["position_m"] for p in placements_from_text(target_json(side(2.9)), room7, ctx7)["placements"]} + assert abs(got["table_1"][0] - 13.6) < 1e-9 and abs(got["cup_1"][0] - 13.6) < 1e-9, got + # a column sunk 0.5 m below the floor is cut at the floor (height 2.3); a box sunk under the floor is not shown + from fastfill.build import FLAT_Z as BUILD_FLAT_Z + assert FLAT_Z == BUILD_FLAT_Z + sunk = [{"id": "c9", "category": "column", "size_xyz_m": [0.3, 0.3, 2.8], "position_m": [12, 22, -0.5]}, + {"id": "u9", "category": "unknown", "size_xyz_m": [1, 1, 0.4], "position_m": [12.5, 22.5, -0.6]}] + r7, c7 = request_to_room({**req, "room": {**req["room"], "fixed_geometry": sunk}, "layout_constraints": []}) + assert [(f["ext_id"], f["pos"][2], f["size"][2]) for f in r7["fixed"]] == [("c9", 0.0, 2.3)], r7["fixed"] + assert {"id": "u9", "reason": "fixed_ignored"} in c7["unsupported"] + # objects are ordered by cm-rounded sizes clamped at 1 cm, as in training (a 3 mm side sorts as 1 cm) + thin = {**req, "objects_to_place": [{"id": "mat", "category": "mat", "size_xyz_m": [1.0, 0.003, 0.9]}, + {"id": "pad", "category": "mat", "size_xyz_m": [0.5, 0.01, 0.9]}], + "layout_constraints": []} + assert [o["ext_id"] for o in request_to_room(thin)[0]["objects"]] == ["mat", "pad"] + # re-entrant corner of an L-shaped room: the nearest-boundary push is diagonal and long, a 6 cm move along -x works + ell = Polygon([(0, 0), (4, 0), (4, 2), (2, 2), (2, 4), (0, 4)]) + plc = {"b": {"pos": [1.76, 2.22, 0], "yaw": 0.0, "on": None}} + snap_inside(plc, {"objects": [{"id": "b", "size": [0.6, 0.6, 0.8]}]}, ell) + assert plc["b"]["pos"][:2] == [1.7, 2.22] or abs(plc["b"]["pos"][0] - 1.7) < 1e-9, plc + # the repair never breaks a hard constraint that held: a chair 3 cm through the south wall faces a box 28.5 deg off + # its front; the shortest push (0, +3 cm) turns that to 32 deg (> 30), so (-4, +3) cm is taken instead + sq, two = Polygon([(0, 0), (4, 0), (4, 3), (0, 3)]), {"objects": [{"id": "c", "size": [.5, .5, .9]}, {"id": "b", "size": [.06, .06, .3]}]} + lay = lambda q: [{"id": i, "pos": q[i]["pos"], "yaw": 0.0, "size": s["size"]} for i, s in zip("cb", two["objects"])] + for cons, want in (([], [1, .25]), ([["faces", "c", "b"]], [.96, .25])): + plc = {"c": {"pos": [1, .22, 0], "yaw": 0.0, "on": None}, "b": {"pos": [1.35, .03, 0], "yaw": 0.0, "on": None}} + assert holds(["faces", "c", "b"], lay(plc), sq.exterior.coords[:-1]) + snap_inside(plc, two, sq, cons, sq.exterior.coords[:-1]) + assert all(abs(u - v) < 1e-9 for u, v in zip(plc["c"]["pos"], want)), plc + assert holds(["faces", "c", "b"], lay(plc), sq.exterior.coords[:-1]) == bool(cons) + # two chairs and a side table between them, all 5 cm through the east wall: the between fails after the second chair + # moves and holds again once the table moves, so the plain pass is kept (a move-by-move guard left chair b outside) + row = {"objects": [{"id": "a", "size": [.7, .7, .9]}, {"id": "b", "size": [.7, .7, .9]}, {"id": "s", "size": [.38, .38, .6]}]} + plc = {i: {"pos": [x, y, 0], "yaw": 0.0, "on": None} for i, x, y in (("a", 3.7, 1), ("b", 3.7, 3), ("s", 3.86, 2))} + sq4 = Polygon([(0, 0), (4, 0), (4, 4), (0, 4)]) + snap_inside(plc, row, sq4, [["between", "s", "a", "b"]], sq4.exterior.coords[:-1]) + assert all(abs(plc[i]["pos"][0] - x) < 1e-9 for i, x in (("a", 3.65), ("b", 3.65), ("s", 3.81))), plc + # a padded keepout ending 3 mm short of the east wall (the finding's 1.48 m room) snaps to it; 2 cm short stays + ko = lambda x1: {"room": {"boundary_xy": [[0, 0], [1.48, 0], [1.48, 3.48], [0, 3.48]]}, + "objects_to_place": [{"id": "a", "category": "chair", "size_xyz_m": [0.5, 0.5, 0.9]}], + "layout_constraints": [{"type": "keepout", "polygon_xy": [[.495, -.2], [x1, -.2], [x1, 1], [.495, 1]]}]} + assert [1.48, 1.08] in request_to_room(ko(1.48 - KEEPOUT_PAD - 0.003))[0]["boundary"] + assert [1.46, 1.08] in request_to_room(ko(1.48 - KEEPOUT_PAD - 0.02))[0]["boundary"] + # two padded keepouts 7.5 mm apart in front of a 1.08 x 0.62 m corner (EmbodiedGen-shaped eg#173): the gap stays open + # and the corner stays floor; 2.9 mm apart the gap folds under cm rounding, as does a keepout crossing a 45-degree + # wall by 3 mm: keepout_not_representable, not bad_boundary (the walls themselves round cleanly) + pair = lambda x1: {"room": {"boundary_xy": [[0, 0], [4.5, 0], [4.5, 3], [0, 3]]}, "objects_to_place": ko(0)["objects_to_place"], + "layout_constraints": [{"type": "keepout", "polygon_xy": [[3.5, 1.1], [4.5, 1.1], [4.5, 2.3], [3.5, 2.3]]}, + {"type": "keepout", "polygon_xy": [[2.1, 2.4], [x1, 2.4], [x1, 3], [2.1, 3]]}]} + assert [4.5, 3] in request_to_room(pair(3.3325))[0]["boundary"] + mouth = {"room": {"boundary_xy": [[0, 0], [4, 0], [4, 4], [1, 4], [0, 3]]}, "objects_to_place": ko(0)["objects_to_place"], + "layout_constraints": [{"type": "keepout", "polygon_xy": [[.58, 2.08], [1.42, 2.08], [1.42, 3.423], [.58, 3.423]]}]} + for b in (pair(3.3371), mouth): + try: + request_to_room(b) + raise AssertionError(b) + except ValueError as e: + assert str(e) == "keepout_not_representable", e + # forms training never has are checked as asked but flagged: faces from a round table (no FRONT head), against_wall + # in a hull room, near from a cup that stands on the table; faces / between / on of the Appendix-B request are trained + assert all(c["trained"] for c in placements_from_text(text, room, ctx)["validation"]["repaired"]["constraints"]) + hull = {"room": {**req["room"], "boundary_source": "scan_hull"}, "objects_to_place": [ + {"id": "rt", "category": "round table", "size_xyz_m": [1, 1, .75]}, {"id": "sh", "category": "shelf", "size_xyz_m": [1, .4, 1.8]}, + {"id": "cup", "category": "cup", "size_xyz_m": [.08, .08, .1]}], "layout_constraints": [ + {"type": "faces", "subject": "rt", "target": "sh"}, {"type": "against_wall", "subject": "sh"}, + {"type": "near", "subject": "cup", "target": "sh", "max_gap_m": 0.5}]} + rh, ch = request_to_room(hull) + ans = '{"placements":[{"id":"round_table_1","pos":[2,2,0],"yaw":270},{"id":"shelf_1","pos":[2,0.2,0],"yaw":0},' \ + '{"id":"cup_1","on":"round_table_1","pos":[2,2,0.75],"yaw":0}]}' # the cup 1.56 m from the shelf + v = placements_from_text(ans, rh, ch)["validation"]["repaired"] + assert [(c["trained"], c["holds"]) for c in v["constraints"]] == [(False, True), (False, True), (False, False)], v + assert not v["ok"] and not v["outside_1mm"] and not v["oob"], v + # build never relates a rug, a sub-5 cm box or (except by 'on') an object standing on another + lay = {"boundary_type": "polygon", "objects": [ + {"id": "s", "category": "sofa", "size": [2, .9, .8]}, {"id": "r", "category": "rug", "size": [2, 1.5, .01]}, + {"id": "l", "category": "lamp", "size": [.3, .3, .5], "parent": "s"}, + {"id": "k", "category": "coaster", "size": [.1, .1, .01], "parent": "s"}]} + assert [trained(k, lay) for k in (["near", "s", "r", .5], ["faces", "s", "r"], ["against_wall", "l"], ["on", "k", "s"], + ["on", "l", "s"], ["against_wall", "s"])] == [False] * 4 + [True] * 2 + print("interface.py self-check ok") + print(json.dumps(out["placements"][2])) diff --git a/code/fastfill/merge_lora.py b/code/fastfill/merge_lora.py new file mode 100644 index 0000000000000000000000000000000000000000..435f5b28da2372e67604cafa9fdc1bba3e2f9b3f --- /dev/null +++ b/code/fastfill/merge_lora.py @@ -0,0 +1,73 @@ +"""Merge a FastFill adapter, retaining its tokenizer and run provenance. + + python -m fastfill.merge_lora --base_model_path BASE --lora_path RUN/final --output_path MERGED + +Uses the local adapter's saved tokenizer when present, otherwise the base tokenizer. +The nearest run_manifest.json selects the run; its numbered resume manifests travel with it. +""" +import argparse +import json +import re +import shutil +from pathlib import Path + +import torch +from peft import PeftModel +from transformers import AutoModelForCausalLM, AutoTokenizer + + +def save_compatibility(output_path): + """Keep the existing Transformers 5 -> 4 artifact compatibility rewrites.""" + cfg_path = Path(output_path) / "config.json" + tok_path = Path(output_path) / "tokenizer_config.json" + cfg = json.loads(cfg_path.read_text()) + tc = json.loads(tok_path.read_text()) + # Transformers 4 needs top-level rope_theta and a list named additional_special_tokens. + theta = (cfg.get("rope_parameters") or {}).get("rope_theta") + compatible_cfg = {"rope_theta": theta, **cfg} if theta is not None else cfg + compatible_tc = ({**{k: v for k, v in tc.items() if k != "extra_special_tokens"}, + "additional_special_tokens": tc["extra_special_tokens"]} + if isinstance(tc.get("extra_special_tokens"), list) else tc) + cfg_path.write_text(json.dumps(compatible_cfg, indent=2) + "\n") + tok_path.write_text(json.dumps(compatible_tc, indent=2, ensure_ascii=False) + "\n") + + +def copy_run_manifests(lora_path, output_path): + """Prefer an adapter-local run; do not mix in another run from its parent.""" + adapter = Path(lora_path) + for run in (adapter, adapter.parent): + if not (run / "run_manifest.json").is_file(): + continue + for manifest in run.iterdir(): + if manifest.is_file() and re.fullmatch(r"run_manifest(?:\.resume[0-9]+)?\.json", manifest.name): + shutil.copyfile(manifest, Path(output_path) / manifest.name) + break + + +def apply_lora(model_name_or_path, output_path, lora_path): + tokenizer_source = lora_path if (Path(lora_path) / "tokenizer_config.json").is_file() else model_name_or_path + tokenizer = AutoTokenizer.from_pretrained(tokenizer_source) + print(f"Loading the base model from {model_name_or_path}") + base = AutoModelForCausalLM.from_pretrained(model_name_or_path, dtype=torch.bfloat16) + print(f"Loading the LoRA adapter from {lora_path}") + lora_model = PeftModel.from_pretrained(base, lora_path, torch_dtype=torch.bfloat16) + print("Applying the LoRA") + model = lora_model.merge_and_unload() + print(f"Saving the target model to {output_path}") + model.save_pretrained(output_path) + tokenizer.save_pretrained(output_path) + save_compatibility(output_path) + copy_run_manifests(lora_path, output_path) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--base_model_path", required=True) + parser.add_argument("--lora_path", required=True) + parser.add_argument("--output_path", required=True) + args = parser.parse_args() + apply_lora(args.base_model_path, args.output_path, args.lora_path) + + +if __name__ == "__main__": + main() diff --git a/code/fastfill/requirements.txt b/code/fastfill/requirements.txt new file mode 100644 index 0000000000000000000000000000000000000000..cb415d6b23920ff9e0c93c00ed76ccbe51a29b98 --- /dev/null +++ b/code/fastfill/requirements.txt @@ -0,0 +1,12 @@ +# FastFill v1. Tested: transformers 4.57.6 (the recorded node03 stack used vLLM 0.19.0), +# 5.14.1 and 5.17.0; peft 0.14-0.21 (4.57.6 + peft 0.21.0 ran the full pipeline). Install torch for the server CUDA first, then: pip install -r fastfill/requirements.txt +# Train, merge and serve in ONE environment. +transformers>=4.56 +peft>=0.14 +accelerate>=1.1 +huggingface_hub>=0.36 +numpy +shapely>=2.0 +ijson +# optional, faster evaluation: vllm +# optional, multi-GPU full fine-tuning: deepspeed diff --git a/code/fastfill/scene.py b/code/fastfill/scene.py new file mode 100644 index 0000000000000000000000000000000000000000..00e86e78b243f1a28ad24ee3e362ca56fe72a825 --- /dev/null +++ b/code/fastfill/scene.py @@ -0,0 +1,379 @@ +"""Canonical room IR and the text protocol FastFill v1 is trained on. + +IR record (one JSON line per room), all lengths in meters, right-handed Z-up, room-local frame +with the boundary AABB min corner at (0, 0): + uid, source, subset, group provenance; `group` = leakage identity (house / scan) + room_type str | None + boundary, boundary_type [[x, y], ...] CCW | None ; "polygon" | "hull" | "proxy" | None + height float | None + objects: [{id, category, size: [sx, sy, sz], pos: [x, y, z], yaw, anchor, parent, desc, tilted}] + size extents along the object's local X/Y/Z (before yaw) + pos bottom center of the object box + yaw radians about +Z, from room +X to the object's local +X = its semantic front + anchor "floor" | "wall" | "ceiling" | "object" | None ; parent = id of supporting object | None + fixed [{id, category, size, pos, yaw}] (build output only): immovable boxes already in the room that the + model is told about but never places (doors, windows, columns, stairs, unlabelled floor boxes) + meta front_known (True only when the adapter verified the front convention), n_incomplete, eval_only, ... + +Model I/O (spec: fastfill I/O design review, 2026-09-24; fixed geometry added 2026-09-24), one turn, compact JSON: + system SYSTEM_PROMPT (constant for training, evaluation and inference) + user {"room_type"?, "boundary_type", "boundary", "height"?, "fixed"?: [{id, size, pos, yaw}], + "objects": [{id, size, desc?}], "constraints"?} + assistant {"placements": [{"id", "on"?, "pos": [x, y, z], "yaw": int deg}, ...]} +Fixed items and objects are listed by footprint area (largest first) with ids slug(category)_k over one shared +namespace; the answer lists the objects only, in their listed order. +Compared with OptiScene: polygon instead of area, one id per instance instead of "N description" groups, +local-axis size with a front convention instead of h/w/d, integer degrees instead of 2-decimal radians, +no design-rule meta prompt, no reasoning block (no reasoning data exists), explicit support ("on"). +""" +import json +import math +import re + +from shapely.geometry import Polygon +from shapely.geometry.polygon import orient + +SYSTEM_PROMPT = ( + "You are a room layout model. Place every listed object in the room; do not add, drop or resize objects.\n" + "Frame: meters, right-handed, Z up, floor z=0. boundary: floor polygon [[x,y],...], counter-clockwise; " + "boundary_type \"polygon\" = walls, \"hull\" = convex hull of a scanned floor (the real floor may be smaller). " + "height: ceiling height. A missing key means unknown.\n" + "fixed: boxes already in the room that cannot move (doors, windows, columns, stairs, ...), each with id, size, " + "pos and yaw as below; keep placements clear of them and do not block doors; they are not part of the answer.\n" + "Object: id, size [sx,sy,sz] along its local X,Y,Z (local +X = its front), optional desc.\n" + "Output only JSON {\"placements\":[{\"id\":id,\"on\":id,\"pos\":[x,y,z],\"yaw\":deg},...]}, " + "every id once, in the listed order. on: the object it stands on, z = that object's top; " + "omit on for the floor, z = 0. pos: bottom center of the box. " + "yaw: integer degrees in [0,360), counter-clockwise from room +X to the object's front.\n" + "constraints (optional, all must hold): [\"on\",a,b]; [\"faces\",a,b] a's front within 30 degrees of the " + "direction to b's center; [\"near\",a,b,d] footprint gap at most d; [\"against_wall\",a] one whole side of " + "a's footprint within 0.1 of a wall; [\"between\",a,b,c] the segment between the centers of b and c crosses a." +) + +HASH = re.compile(r"(?=[a-f]*\d)[0-9a-f]{8}\b") # SAGE asset hashes: glassnightstand5c8f77c6 +GENERIC = re.compile(r"^(other\w*|objects?|unknown)( |$)") # labels that say nothing: shown as "object" +UNKNOWN_ROOM = {"misc", "other", "other room", "unknown", "undefined", "none"} +REL = re.compile(r"[,;]?\s*\b(positioned|placed|resting|located|situated|sitting on|standing on|lying on|leaning " + r"against|next to|beside|in front of|on top of|on a flat surface|on the floor)\b.*$", re.I) +DANGLING = {"a", "an", "the", "and", "or", "with", "of", "in", "on", "to", "for", "at", "by", "featuring", "its"} + + +# labels whose first synonym is a bare modifier ('corner', 'three seat') or an Infinigen factory name +ALIASES = {"corner/side table": "side table", "three-seat / multi-seat sofa": "three seat sofa", + "three-seat / multi-person sofa": "three seat sofa", "coffeetablefactory": "coffee table", + "sidetablefactory": "side table", "boxcomforterfactory": "comforter", "comforterfactory": "comforter"} + + +def norm_cat(c): + """Traceable category normalization (raw label stays in IR): first synonym, words only, no hash/instance no.""" + c = (c or "").lower() + c = ALIASES.get(re.sub(r"\(.*", "", c).strip(), c) + c = re.split(r"[,(/]", c)[0] + c = " ".join(re.sub(r"[^a-z0-9]+", " ", c).split()) + c = " ".join(HASH.sub(" ", c).split()).split(" or ")[0] + return re.sub(r"(?<=[a-z])(?:\s*\d+)+$", "", c) + + +def head(o): + """Head noun of an object's category (last word), used by anchor rules and validator buckets.""" + w = norm_cat(o["category"]).split() + return w[-1] if w else "" + + +ROOM_WORDS = {"livingroom": "living room", "diningroom": "dining room", "livingdiningroom": "living dining room", + "masterbedroom": "master bedroom", "secondbedroom": "second bedroom", "kidsroom": "kids room", + "storageroom": "storage room", "laundryroom": "laundry room", "elderlyroom": "elderly room"} + + +def norm_room_type(t): + if not t: + return None + t = re.sub(r"\s*/\s*", "/", " ".join(t.lower().replace("_", " ").split())).rstrip(".").strip() + t = " ".join(ROOM_WORDS.get(w, w) for w in t.split()) + t = re.sub(r"(?:\s+\d+)+$", "", t) # "meeting room 1" -> "meeting room" + # floor / apartment / capacity tags of MansionWorld: "restroom f1", "apartment 2 storage", "meeting room 4p" + t = " ".join(re.sub(r"\b(?:f\d+|apt\d+|apartment \d+|\d+p)\b", " ", t).split()) + t = re.sub(r"\brestrooms\b", "restroom", t) + return None if not t or t in UNKNOWN_ROOM else t + + +def num(x): + """2-decimal number as shown to the model; integral values without '.0', no '-0'.""" + v = round(float(x), 2) + 0.0 + return int(v) if v == int(v) else v + + +def shown_size(o): + return [max(num(v), 0.01) for v in o["size"]] + + +def short_desc(d, max_words=16): + """First sentence, no article, no placement clause (scene context, not asset), cut at a clause boundary.""" + s = re.split(r"(?<=[a-z])\.\s", d.strip())[0].rstrip(".") + s = re.sub(r"^(a|an|the)\s+", "", s, flags=re.I) + w = REL.sub("", s).strip(" ,;").split() + if len(w) > max_words: + w = w[:max_words] + cut = [i for i, x in enumerate(w) if i >= 4 and (x.endswith(",") or x.lower() in ("with", "featuring", "and"))] + if cut: + w = w[:cut[-1] + 1] if w[cut[-1]].endswith(",") else w[:cut[-1]] + while w and w[-1].lower().strip(",") in DANGLING: + w.pop() + return " ".join(w).strip(" ,;") + + +def _clean_once(b): + p = Polygon([(round(x, 2), round(y, 2)) for x, y in b]).simplify(0.01, preserve_topology=True) + if not p.is_valid and p.area > 0: # cm rounding can pinch a valid ring into a self-touch: repair if harmless + q = p.buffer(0) + if q.geom_type == "Polygon" and not q.interiors and abs(q.area - p.area) < 0.01 * p.area: + p = q + if not p.is_valid or p.area <= 0: + return None + pts = list(orient(p, 1.0).exterior.coords)[:-1] + k = min(range(len(pts)), key=lambda i: (pts[i][0] ** 2 + pts[i][1] ** 2, pts[i])) + return [[num(x), num(y)] for x, y in pts[k:] + pts[:k]] + + +def clean_boundary(b): + """cm rounding, duplicate/collinear removal (<=1 cm deviation), CCW, start at the vertex nearest (0,0). + Repeated until nothing changes (simplify never drops a ring's start vertex, and the start moves), so it is + idempotent: a request echoing a clean boundary keeps it.""" + out = _clean_once(b) + for _ in range(8): + nxt = _clean_once(out) if out else None + if nxt is None or nxt == out: + break + out = nxt + return out + + +def rot90(room, k): + """Exact augmentation on cm-rounded rooms: rotate about the origin by k*90 deg, shift back to the min corner. + Run BEFORE canonical.""" + if k % 4 == 0: + return room + c, s = [(1, 0), (0, 1), (-1, 0), (0, -1)][k % 4] + rot = lambda x, y: (c * x - s * y, s * x + c * y) + b = [rot(x, y) for x, y in room["boundary"]] + mx, my = min(p[0] for p in b), min(p[1] for p in b) + b = [[num(x - mx), num(y - my)] for x, y in b] # rotation keeps CCW; only the start vertex moves + s0 = min(range(len(b)), key=lambda i: (b[i][0] ** 2 + b[i][1] ** 2, b[i])) + def turn(o): + x, y = rot(*o["pos"][:2]) + return {**o, "pos": [num(x - mx), num(y - my), o["pos"][2]], "yaw": (o["yaw"] + k * math.pi / 2) % (2 * math.pi)} + out = {**room, "boundary": b[s0:] + b[:s0], "objects": [turn(o) for o in room["objects"]]} + if room.get("fixed"): + out["fixed"] = [turn(o) for o in room["fixed"]] + return out + + +def slug(category): + nc = norm_cat(category) + return "object" if not nc or GENERIC.match(nc) else nc.replace(" ", "_") + + +def canonical(room): + """Order by footprint area desc (ties: category, size); identical instances by (z, x, y) when poses exist + (training), else keep input order (inference; the sort is stable). ids become slug(norm_cat)_k, counted over + the fixed items first, then the objects, so no id is shared between the two lists.""" + def key(o): + s = [max(round(v, 2), 0.01) for v in o["size"]] # = shown_size (training sizes are already shown_size) + k = (-round(s[0] * s[1], 4), norm_cat(o["category"]), [-v for v in s]) + return k + ((round(o["pos"][2], 2), round(o["pos"][0], 2), round(o["pos"][1], 2)) if "pos" in o else ()) + fixed = sorted(room.get("fixed") or [], key=key) + objs = sorted(room["objects"], key=key) + count, new = {}, {} + for o in fixed + objs: + sl = slug(o["category"]) + count[sl] = count.get(sl, 0) + 1 + new[o["id"]] = f"{sl}_{count[sl]}" + out = {**room, "objects": [{**o, "id": new[o["id"]], "parent": new.get(o.get("parent"))} for o in objs]} + if fixed: + out["fixed"] = [{**o, "id": new[o["id"]]} for o in fixed] + return out + + +def user_json(room, constraints=None, with_desc=True): + d = {} + rt = norm_room_type(room.get("room_type")) + if rt: + d["room_type"] = rt + d["boundary_type"] = room["boundary_type"] + d["boundary"] = room["boundary"] + if room.get("height"): + d["height"] = num(room["height"]) + if room.get("fixed"): + d["fixed"] = [{"id": o["id"], "size": shown_size(o), "pos": [num(v) for v in o["pos"]], + "yaw": int(round(math.degrees(o["yaw"]))) % 360} for o in room["fixed"]] + objs = [] + for o in room["objects"]: + e = {"id": o["id"], "size": shown_size(o)} + desc = short_desc(o["desc"]) if with_desc and o.get("desc") else "" + if desc: # a desc that is only a placement clause shortens to "" + e["desc"] = desc + objs.append(e) + d["objects"] = objs + if constraints: + d["constraints"] = constraints + return json.dumps(d, ensure_ascii=False, separators=(",", ":")) + + +def target_json(room): + out = [] + for o in room["objects"]: + p = {"id": o["id"]} + if o.get("parent"): + p["on"] = o["parent"] + p["pos"] = [num(v) for v in o["pos"]] + p["yaw"] = int(round(math.degrees(o["yaw"]))) % 360 + out.append(p) + return json.dumps({"placements": out}, ensure_ascii=False, separators=(",", ":")) + + +def messages(room, constraints=None, with_desc=True, with_target=True): + m = [{"role": "system", "content": SYSTEM_PROMPT}, + {"role": "user", "content": user_json(room, constraints, with_desc)}] + if with_target: + m.append({"role": "assistant", "content": target_json(room)}) + return m + + +def prompt_text(tok, msgs): + """Chat-template prompt ending right where the assistant answer starts (Qwen3 thinking off).""" + return tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False) + + +def _isnum(v): + return isinstance(v, (int, float)) and not isinstance(v, bool) + + +MAX_COORD = 1e3 + + +def parse(text, ids): + """Never raises. -> (placements {id: {pos, yaw (rad), on}}, error | None). + error: truncated_or_bad_json | bad_json | bad_item | bad_number | duplicate_id | missing_ids | extra_ids | + bad_support_ref (unknown, self or cyclic 'on').""" + s = text.split("", 1)[-1] if isinstance(text, str) else "" + i = s.find("{") + try: + items = json.JSONDecoder().raw_decode(s, i)[0]["placements"] if i >= 0 else None + except (ValueError, KeyError, TypeError, IndexError, RecursionError): + return {}, "truncated_or_bad_json" + if not isinstance(items, list): + return {}, "bad_json" + out, err = {}, None + for p in items: + pos, yaw, on = (p.get("pos"), p.get("yaw"), p.get("on")) if isinstance(p, dict) else (None, None, None) + if not (isinstance(p, dict) and isinstance(p.get("id"), str) and isinstance(pos, list) and len(pos) == 3 + and all(map(_isnum, pos)) and _isnum(yaw) and (on is None or isinstance(on, str))): + err = err or "bad_item" + continue + try: + vals = [float(v) for v in pos + [yaw]] # OverflowError on absurdly long integers + except OverflowError: + vals = [math.inf] + if not all(math.isfinite(v) and abs(v) < MAX_COORD for v in vals): # 1 km: nothing in a room is further + err = err or "bad_number" + continue + if p["id"] in out: + err = err or "duplicate_id" + continue + out[p["id"]] = {"pos": vals[:3], "yaw": math.radians(vals[3] % 360), "on": on} + if err is None and set(out) != set(ids): + err = "missing_ids" if set(ids) - set(out) else "extra_ids" + if err is None: + for pid, p in out.items(): + seen, cur = {pid}, p["on"] + while cur is not None: + if cur not in out or cur in seen: + return out, "bad_support_ref" + seen.add(cur) + cur = out[cur]["on"] + return out, err + + +def gt_placements(room): + """Reference layout in parse() form (evaluate --gt-only).""" + return {o["id"]: {"pos": o["pos"], "yaw": o["yaw"], "on": o.get("parent")} for o in room["objects"]} + + +def apply(room, pl): + """Predicted layout in IR form; parents come ONLY from the prediction; missing objects are dropped.""" + return {**room, "objects": [{**o, "pos": pl[o["id"]]["pos"], "yaw": pl[o["id"]]["yaw"], + "parent": pl[o["id"]]["on"]} for o in room["objects"] if o["id"] in pl]} + + +def footprint(o): + """Oriented footprint corners [(x, y)] x4 of an upright object.""" + c, s = math.cos(o["yaw"]), math.sin(o["yaw"]) + hx, hy = o["size"][0] / 2, o["size"][1] / 2 + x, y = o["pos"][0], o["pos"][1] + return [(x + c * dx - s * dy, y + s * dx + c * dy) for dx, dy in ((hx, hy), (-hx, hy), (-hx, -hy), (hx, -hy))] + + +if __name__ == "__main__": + assert norm_cat("toilet, can, commode") == "toilet" and norm_cat("ceiling lamp(traffic light)") == "ceiling lamp" + assert norm_cat("pen_1") == "pen" == norm_cat("pen2") and norm_cat("2 seat sofa") == "2 seat sofa" + assert norm_cat("Lounge Chair / Cafe Chair") == "lounge chair" and norm_cat("King-size Bed") == "king size bed" + assert norm_cat("glassnightstand5c8f77c6") == "glassnightstand" and norm_cat("plant_or_flower_pot") == "plant" + assert norm_room_type("Bedroom / Hotel") == "bedroom/hotel" and norm_room_type("Misc.") is None + assert norm_room_type("LivingRoom") == "living room" == norm_room_type("living_room") + assert norm_room_type("Meeting Room 1") == "meeting room" + assert parse('{"placements":[{"id":"a","pos":[' + '9' * 400 + ',1,0],"yaw":0}]}', ["a"])[1] == "bad_number" + assert short_desc("A wooden table positioned next to a rectangular object. It is old.") == "wooden table" + assert num(0.0) == 0 and num(-0.001) == 0 and num(2.704) == 2.7 + assert clean_boundary([[0, 0], [2, 0], [4, 0], [4, 3], [4, 3], [0, 3]]) == [[0, 0], [4, 0], [4, 3], [0, 3]] + wob = [[0, 0], [2.004, 0.006], [4, 0], [4.006, 1.504], [4, 3], [0, 3]] # near-collinear after cm rounding + assert clean_boundary(clean_boundary(wob)) == clean_boundary(wob) + + ok = '{"placements":[{"id":"desk_1","pos":[1,1,0],"yaw":90},{"id":"lamp_1","on":"desk_1","pos":[1,1,0.75],"yaw":0}]}' + ids = ["desk_1", "lamp_1"] + pl, err = parse("\n\n\n\n" + ok + "\nnote {x}", ids) + assert err is None and abs(pl["desk_1"]["yaw"] - math.pi / 2) < 1e-9 and pl["lamp_1"]["on"] == "desk_1" + for bad, want in (('"on":"lamp_1"', "bad_support_ref"), ('"on":"sofa_1"', "bad_support_ref"), + ('"on":["desk_1"]', "bad_item"), ('"on":{"id":"desk_1"}', "bad_item")): + assert parse(ok.replace('"on":"desk_1"', bad), ids)[1] == want, bad + assert parse(ok.replace('"yaw":0}', '"yaw":NaN}'), ids)[1] == "bad_number" + assert parse(ok.replace('"yaw":90', '"yaw":true'), ids)[1] == "bad_item" + assert parse(ok.replace('"pos":[1,1,0]', '"pos":"110"'), ids)[1] == "bad_item" + assert parse('{"placements":' + "[" * 30000 + "]" * 30000 + "}", ids)[1] in ("truncated_or_bad_json", "bad_item") + assert parse(ok[:70], ids)[1] == "truncated_or_bad_json" and parse(None, ids)[1] == "bad_json" + assert parse('{"placements":[{"id":"desk_1","pos":[1,1,0],"yaw":0}]}', ids)[1] == "missing_ids" + + room = {"room_type": "Bed_Room", "boundary_type": "polygon", "boundary": [[0, 0], [4, 0], [4, 3], [0, 3]], "height": 2.7, + "objects": [{"id": s, "category": "chair", "size": [.5, .5, .9], "pos": [x, y, 0], "yaw": 0} + for s, x, y in (("a", 1, 1), ("b", 3, 1), ("c", 2, 2))] + + [{"id": "d", "category": "desk", "size": [1.2, .6, .75], "pos": [2, .3, 0], "yaw": math.pi / 2}, + {"id": "l", "category": "lamp", "size": [.2, .2, .4], "pos": [2, .3, .75], "yaw": 0, "parent": "d"}]} + r4 = room + for k in range(4): # rot90 before canonical keeps the (z, x, y) numbering of identical chairs + c = canonical(rot90(room, k)) + chairs = [o for o in c["objects"] if o["category"] == "chair"] + assert [o["pos"][:2] for o in chairs] == sorted(o["pos"][:2] for o in chairs), k + r4 = rot90(r4, 1) + assert r4["boundary"] == room["boundary"] and [o["pos"] for o in r4["objects"]] == [o["pos"] for o in room["objects"]] + c = canonical(room) + assert [o["id"] for o in c["objects"]] == ["desk_1", "chair_1", "chair_2", "chair_3", "lamp_1"] + m = messages(c) + assert json.loads(m[1]["content"])["room_type"] == "bed room" and "fixed" not in json.loads(m[1]["content"]) + # fixed boxes: shared id namespace (a fixed chair takes chair_1), shown with pose, rotated with the room, not answered + fx = {**room, "fixed": [{"id": "col", "category": "column", "size": [.3, .3, 2.7], "pos": [3.5, 2.5, 0], "yaw": 0}, + {"id": "fc", "category": "chair", "size": [.5, .5, .9], "pos": [.5, .5, 0], "yaw": 0}, + {"id": "u", "category": "unknown", "size": [1, .4, .8], "pos": [2, 2.8, 0], "yaw": math.pi / 2}]} + cf = canonical(fx) + assert [o["id"] for o in cf["fixed"]] == ["object_1", "chair_1", "column_1"], cf["fixed"] + assert [o["id"] for o in cf["objects"]] == ["desk_1", "chair_2", "chair_3", "chair_4", "lamp_1"] + uj = json.loads(user_json(cf)) + assert uj["fixed"][1] == {"id": "chair_1", "size": [0.5, 0.5, 0.9], "pos": [0.5, 0.5, 0], "yaw": 0} and "fixed" not in target_json(cf) + r1 = rot90(fx, 1) # (3.5, 2.5) turned by 90 deg about the origin -> (-2.5, 3.5) -> shifted + assert r1["fixed"][0]["pos"] == [0.5, 3.5, 0] and abs(r1["fixed"][0]["yaw"] - math.pi / 2) < 1e-9, r1["fixed"][0] + assert slug("otherprop") == "object" == slug("") == slug("Objects") and slug("floor lamp") == "floor_lamp" + pl, err = parse(m[2]["content"], [o["id"] for o in c["objects"]]) + assert err is None and pl["lamp_1"]["on"] == "desk_1" and pl["lamp_1"]["pos"] == [2.0, 0.3, 0.75] + assert apply(c, pl)["objects"][-1]["parent"] == "desk_1" + fp = footprint(c["objects"][0]) # desk rotated 90 deg: 0.6 wide in x, 1.2 deep in y + xs, ys = [p[0] for p in fp], [p[1] for p in fp] + assert abs(max(xs) - min(xs) - 0.6) < 1e-9 and abs(max(ys) - min(ys) - 1.2) < 1e-9 + print("scene.py self-check ok") diff --git a/code/fastfill/serve.py b/code/fastfill/serve.py new file mode 100644 index 0000000000000000000000000000000000000000..fb34d368971412f5419c3553f622d370c2ae9d9b --- /dev/null +++ b/code/fastfill/serve.py @@ -0,0 +1,188 @@ +"""OpenAI-compatible endpoint that speaks the WorldEdge request (scope doc Appendix B, fields in interface.py). + + python -m fastfill.serve --model outputs/ff-v1-merged --port 8001 + # EmbodiedGen side: WORLDEDGE_FASTFILL_URL=http://:8001/v1 WORLDEDGE_FASTFILL_MODEL=ff-v1-merged + python -m fastfill.serve # self-check (no model) + +The last user message is the request JSON (the body's "model" field is not read). The reply's content is +placements_from_text's JSON ({"placements": [{"id", "position_m", "yaw_rad", "support_parent", "support_surface"?}], "raw_placements": [...], +"validation": {"raw", "repaired"}, "error": null, "unsupported": [...]}) plus "model" (--name, default the checkpoint +directory's name) and "checkpoint" (its absolute path): EmbodiedGen caches that content under its own +WORLDEDGE_FASTFILL_MODEL, so set that to the served name, and a cached layout still says which checkpoint made it. +A request with nothing to place (no objects, or all anchored to a wall or ceiling) is answered 200 with no placements +and its unsupported list, without calling the model (training never has an empty room). +HTTP 422 and no layout when the request is rejected (a prompt of --max_len tokens or more included: prompt_too_long), +the answer does not parse (a missing or extra object included), or the repaired layout still fails the canonical +checks: FastFill places every object validly or fails back to the caller, it never drops one. Every error body carries +model and checkpoint; a 422 after generation also the model's text (raw_text) and the unsupported list. +Greedy decoding, one request at a time. +The JSON body requires a positive Content-Length of at most MAX_REQUEST_BYTES (1 MiB). Missing/invalid lengths +and malformed or excessively nested JSON receive 400; oversized bodies receive 413 before any body read. +""" +import argparse +import json +import os +import sys +import time +from http.server import BaseHTTPRequestHandler, HTTPServer + +from fastfill.interface import placements_from_text, request_to_room +from fastfill.scene import messages, prompt_text + +MAX_REQUEST_BYTES = 1024 * 1024 + + +class PromptTooLong(ValueError): + """Raised by generate() for a prompt of max_len tokens or more (vLLM cannot start on it): answered with 422.""" + + +def answer(req, generate): + """Request dict -> (HTTP status, body). generate(model messages) -> model text, or raises PromptTooLong.""" + try: + room, ctx = request_to_room(req) + except ValueError as e: + return 422, {"error": {"message": str(e), "type": "invalid_request_error"}} + if not room["objects"]: # the empty answer is the only valid one: not generated + return 200, placements_from_text('{"placements":[]}', room, ctx) + try: + text = generate(messages(room, room["constraints"], with_target=False)) + except PromptTooLong as e: + return 422, {"error": {"message": str(e), "type": "invalid_request_error", "unsupported": ctx["unsupported"]}} + out = placements_from_text(text, room, ctx) + if out["error"]: + return 422, {"error": {"message": f"model answer rejected: {out['error']}", "type": "invalid_layout", + "raw_text": text, "unsupported": out["unsupported"]}} + if not out["validation"]["repaired"]["ok"]: # every object must be placed validly; else the caller decides + return 422, {"error": {"message": "layout fails validation even after repair", "type": "invalid_layout", + "raw_placements": out["raw_placements"], "validation": out["validation"], + "raw_text": text, "unsupported": out["unsupported"]}} + return 200, out + + +def make_handler(generate, name, checkpoint): + class Handler(BaseHTTPRequestHandler): + def do_POST(self): + if not self.path.endswith("/chat/completions"): + return self.reply(404, {"error": {"message": "POST .../v1/chat/completions"}}) + try: + length = int(self.headers["Content-Length"]) + if length <= 0: + raise ValueError("Content-Length must be positive") + if length > MAX_REQUEST_BYTES: + return self.reply(413, {"error": {"message": f"body exceeds {MAX_REQUEST_BYTES} bytes", + "type": "invalid_request_error"}}) + body = json.loads(self.rfile.read(length)) + req = json.loads(body["messages"][-1]["content"]) + except (ValueError, KeyError, IndexError, TypeError, RecursionError) as e: + return self.reply(400, {"error": {"message": f"bad body: {e!r}", "type": "invalid_request_error"}}) + try: + status, out = answer(req, generate) + except Exception as e: # a bug or a dead model: say so instead of dropping the connection + return self.reply(500, {"error": {"message": repr(e), "type": "server_error"}}) + if status == 200: + content = json.dumps({**out, "model": name, "checkpoint": checkpoint}) + out = {"id": f"fastfill-{time.time_ns()}", "object": "chat.completion", "created": int(time.time()), + "model": name, "choices": [{"index": 0, "finish_reason": "stop", + "message": {"role": "assistant", "content": content}}], + "usage": {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}} + self.reply(status, out) + + def reply(self, status, obj): + if status != 200: + obj["error"].update(model=name, checkpoint=checkpoint) + data = json.dumps(obj).encode() + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + + return Handler + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--model", required=True, help="merged checkpoint dir") + ap.add_argument("--name", help="model name echoed in replies (default: the checkpoint directory's name)") + ap.add_argument("--host", default="0.0.0.0") + ap.add_argument("--port", type=int, default=8001) + ap.add_argument("--max_len", type=int, default=40960, help="model sequence limit; the answer may use what the prompt leaves") + a = ap.parse_args() + from vllm import LLM, SamplingParams + llm = LLM(model=a.model, max_model_len=a.max_len) + tok = llm.get_tokenizer() + + def generate(msgs): + p = prompt_text(tok, msgs) + n = len(tok(p)["input_ids"]) + if n >= a.max_len: # vLLM raises on such a prompt: refuse it as a bad request instead + raise PromptTooLong(f"prompt_too_long: {n} tokens, limit {a.max_len}") + sp = SamplingParams(temperature=0, max_tokens=a.max_len - n) + return llm.generate([p], sp, use_tqdm=False)[0].outputs[0].text + ckpt = os.path.abspath(a.model) if os.path.isdir(a.model) else a.model # a dir, or a hub id as given + name = a.name or os.path.basename(os.path.normpath(ckpt)) + print(f"FastFill endpoint: http://{a.host}:{a.port}/v1 model {name} = {ckpt}", flush=True) + HTTPServer((a.host, a.port), make_handler(generate, name, ckpt)).serve_forever() + + +if __name__ == "__main__": + if sys.argv[1:]: + main() + else: # self-check: answer() and the HTTP wrapper, a stub model + import threading + import urllib.error + import urllib.request + calls = [] + + def stub(text): + def gen(msgs): + calls.append(json.loads(msgs[1]["content"])) + if text is None: + raise PromptTooLong("prompt_too_long: 50000 tokens, limit 40960") + return text + return gen + req = {"room": {"boundary_xy": [[0, 0], [4, 0], [4, 4], [0, 4]], "height_m": 2.8, + "fixed_geometry": [{"id": "col_1", "category": "column", "size_xyz_m": [0.3, 0.3, 2.8], + "position_m": [3.5, 3.5, 0]}]}, + "objects_to_place": [{"id": "table_1", "category": "table", "size_xyz_m": [1.4, 0.8, 0.75]}, + {"id": "cup_1", "category": "cup", "size_xyz_m": [0.08, 0.08, 0.1]}, + {"id": "clock_1", "category": "clock", "size_xyz_m": [0.3, 0.05, 0.3], "anchor": "wall"}]} + good = '{"placements":[{"id":"table_1","pos":[2,2,0],"yaw":0},{"id":"cup_1","on":"table_1","pos":[2,2,0.75],"yaw":0}]}' + clock = [{"id": "clock_1", "reason": "unsupported_anchor"}] + st, out = answer(req, stub(good)) + assert st == 200 and len(calls) == 1 and out["validation"]["repaired"]["ok"] and out["unsupported"] == clock, out + # nothing placeable: no objects, or only wall / ceiling items -> 200, empty placements, the model never called + for objs, unsup in (([], []), (req["objects_to_place"][2:], clock)): + st, out = answer({**req, "objects_to_place": objs}, stub("never")) + assert st == 200 and len(calls) == 1 and out["placements"] == [] and out["unsupported"] == unsup, out + assert out["error"] is None and out["validation"]["repaired"]["ok"], out + # 422 after generation keeps the model's text and the unsupported list: bad JSON, a missing object, a layout + # that fails after repair (table through the column), a prompt over the limit (no text) + on_col = good.replace("[2,2,0]", "[3.4,3.4,0]").replace("[2,2,0.75]", "[3.4,3.4,0.75]") + for text, kind in (("{not json", "invalid_layout"), (good.split(',{"id":"cup_1"')[0] + "]}", "invalid_layout"), + (on_col, "invalid_layout"), (None, "invalid_request_error")): + st, out = answer(req, stub(text)) + e = out["error"] + assert st == 422 and e["type"] == kind and e["unsupported"] == clock and e.get("raw_text") == text, out + assert out["error"]["message"].startswith("prompt_too_long") + assert answer(req, stub(on_col))[1]["error"]["validation"]["repaired"]["fixed_blocking"] == [["table_1", "col_1"]] + # over HTTP: every reply names the served model and checkpoint (200 in the content EmbodiedGen caches) + texts = [good, "{not json"] + srv = HTTPServer(("127.0.0.1", 0), make_handler(lambda m: texts.pop(0), "ff-test", "/ckpt/ff-test")) + threading.Thread(target=srv.serve_forever, daemon=True).start() + + def post(obj): + body = json.dumps({"model": "fastfill", "messages": [{"role": "user", "content": json.dumps(obj)}]}).encode() + try: + with urllib.request.urlopen(f"http://127.0.0.1:{srv.server_port}/v1/chat/completions", body) as f: + return f.status, json.load(f) + except urllib.error.HTTPError as f: + return f.code, json.load(f) + st, rep = post(req) + content = json.loads(rep["choices"][0]["message"]["content"]) + assert st == 200 and rep["model"] == "ff-test" and (content["model"], content["checkpoint"]) == ("ff-test", "/ckpt/ff-test") + st, rep = post(req) + assert st == 422 and rep["error"]["model"] == "ff-test" and rep["error"]["raw_text"] == "{not json", rep + assert post({"room": {}})[0] == 422 and post({"room": {}})[1]["error"]["checkpoint"] == "/ckpt/ff-test" + srv.shutdown() + print("serve.py self-check ok") diff --git a/code/fastfill/split.py b/code/fastfill/split.py new file mode 100644 index 0000000000000000000000000000000000000000..a0c28ffd087a1c4361ee4b6643d87dad6ef19a90 --- /dev/null +++ b/code/fastfill/split.py @@ -0,0 +1,106 @@ +"""Leakage-safe train/dev/test assignment. + +Rooms are linked when they share + - a `group` (same house / scan across exports), + - a meta.group_aliases entry (e.g. several scans of one ARKit visit), + - a furniture content key (3D-FRONT reuses whole furniture sets across houses; exports repackage rooms), + - a layout key (same boundary and object poses up to a 90-degree rotation, whatever the assets). +Each connected component goes to one split, chosen by a stable hash; a component holding an eval-only room +(meta.eval_only) goes to test as a whole. +""" +import hashlib +import math + +from fastfill.scene import num, rot90 + +# Procedurally generated sources share one asset library across all plans: equal furniture sets there are not +# repackaged rooms, and linking them fuses thousands of rooms into one component. +GROUP_ONLY = {"MansionWorld"} + + +def content_key(room, step=0.05): + """Rotation/translation-invariant fingerprint: multiset of sorted object dims (5 cm grid).""" + if len(room["objects"]) < 3 or room.get("source") in GROUP_ONLY: + return None + dims = sorted(tuple(sorted(round(v / step) for v in o["size"])) for o in room["objects"]) + return hashlib.sha1(repr(dims).encode()).hexdigest() + + +def layout_key(room): + """Asset-independent design fingerprint: boundary + poses (cm, integer degrees), min over 4 rotations.""" + if not room.get("boundary"): + return None + keys = [] + for k in range(4): + r = rot90(room, k) + poses = sorted((num(o["pos"][0]), num(o["pos"][1]), round(math.degrees(o["yaw"])) % 360) for o in r["objects"]) + keys.append(hashlib.sha1(repr((r["boundary"], poses)).encode()).hexdigest()) + return min(keys) + + +def assign_splits(rooms, dev=0.05, test=0.05): + """rooms: list of IR dicts -> list of 'train' | 'dev' | 'test' (same order).""" + parent = {} + + def find(x): + while parent.setdefault(x, x) != x: + parent[x] = parent[parent[x]] + x = parent[x] + return x + + def union(a, b): + ra, rb = find(a), find(b) + if ra != rb: + parent[max(ra, rb)] = min(ra, rb) + + for r in rooms: + g = "g:" + r["group"] + find(g) + for key in ("c:%s" % content_key(r), "l:%s" % layout_key(r)): + if not key.endswith("None"): + union(g, key) + for al in r.get("meta", {}).get("group_aliases") or []: + union(g, "a:" + al) + held_out = {find("g:" + r["group"]) for r in rooms if r.get("meta", {}).get("eval_only")} + # a component is hashed by its smallest GROUP name, not by its union-find root (which may be a furniture or layout + # key): the split of a house then does not depend on object content, and a prep change does not move houses + name = {} + for r in rooms: + c = find("g:" + r["group"]) + name[c] = min(name.get(c, "g:" + r["group"]), "g:" + r["group"]) + out = [] + for r in rooms: + c = find("g:" + r["group"]) + h = int(hashlib.sha1(name[c].encode()).hexdigest(), 16) % 10000 / 10000 + out.append("test" if c in held_out or h < test else "dev" if h < test + dev else "train") + return out + + +if __name__ == "__main__": + def room(group, sizes, poses=None, boundary=((0, 0), (5, 0), (5, 4), (0, 4)), source="S", meta=None): + poses = poses or [(1 + i, 1, 0) for i in range(len(sizes))] + return {"group": group, "source": source, "boundary": [list(p) for p in boundary], "meta": meta or {}, + "objects": [{"size": s, "pos": [x, y, 0], "yaw": math.radians(d)} for s, (x, y, d) in zip(sizes, poses)]} + + a = room("house1", [[2, 1.6, .5], [.5, .5, .6], [.5, .5, .6]]) + b = room("house2", [[1.6, 2, .5], [.5, .5, .6], [.5, .6, .5]]) # same furniture, other house, axes permuted + c = room("house1", [[1, 1, 1]]) # same house, different room + v1 = room("vid1", [[1, 2, 3]], meta={"group_aliases": ["visit:9"]}) + v2 = room("vid2", [[3, 2, 1.5]], meta={"group_aliases": ["visit:9"]}) # two videos of one visit + t1 = room("h_t1", [[1, 1, 1], [2, 2, 2]], [(1, 1, 0), (3, 2, 90)]) + t2 = room("h_t2", [[.9, .9, 1.2], [2, 2, 1]], [(3, 1, 90), (2, 3, 180)], # same design stored rotated, new assets + boundary=((0, 0), (4, 0), (4, 5), (0, 5))) + e1 = room("eval1", [[1, 1, 1], [1, 2, 1], [2, 2, 2]], meta={"eval_only": True}) + e2 = room("twin", [[1, 1, 1], [2, 1, 1], [2, 2, 2]]) # content twin of an eval-only room + m1 = room("mw1", [[1, 1, 1]] * 3, source="MansionWorld") + m2 = room("mw2", [[1, 1, 1]] * 3, [(2, 2, 0), (3, 2, 0), (4, 3, 0)], source="MansionWorld") + rooms = [a, b, c, v1, v2, t1, t2, e1, e2, m1, m2] + [ + room(f"h{i}", [[i / 10 + .1, 1, 1]] * 3, [(1 + i / 1000, 1, 0), (2, 2, 0), (3, 3, 0)]) for i in range(2000)] + s = assign_splits(rooms) + assert s[0] == s[1] == s[2] and s[3] == s[4] and s[5] == s[6], s[:7] + assert s[7] == s[8] == "test", s[7:9] + assert content_key(m1) is None and layout_key(m1) != layout_key(m2) and layout_key(t1) == layout_key(t2) + frac = {k: s.count(k) / len(s) for k in ("train", "dev", "test")} + assert 0.85 < frac["train"] < 0.95 and 0.02 < frac["test"] < 0.08, frac + assert assign_splits(rooms) == s + print("split.py self-check ok", frac) diff --git a/code/fastfill/tests/test_adapter_repairs.py b/code/fastfill/tests/test_adapter_repairs.py new file mode 100644 index 0000000000000000000000000000000000000000..a8a447367d3628100fcc96d17b8f73f84cde4871 --- /dev/null +++ b/code/fastfill/tests/test_adapter_repairs.py @@ -0,0 +1,491 @@ +"""Bounded adapter regressions; fixtures are self-contained, with no corpus/audit files read. + +The 21 source boxes below are the complete retained >2 cm error examples in the +2026-09-27 adapter evidence (11 Structured3D, 8 HSSD200, 2 MultiScan). +""" +import copy +import csv +import io +import itertools +import json +import math +from types import SimpleNamespace +from unittest.mock import patch + +import numpy as np +import pytest + +from fastfill.adapters import hssd200, interiorgs, multiscan, structured3d +from fastfill.adapters.unified import convert_room +from fastfill.build import prep + + +AUDITED_BOXES = [('Structured3D', + 'structured3d_00160_room_584/50', + {'category': 'unknown', + 'instance_id': 50, + 'position': {'x': 1.516548, 'y': 0.92628, 'z': 0.4325}, + 'raw_basis': [[-0.9999999999999962, 8.742277644230737e-08, 0.0], + [3.8213709261743826e-15, 4.3711388286737763e-08, -0.9999999999999991], + [-8.742277644230729e-08, -0.9999999999999953, -4.3711388175715626e-08]], + 'raw_coeffs': [208.00000000000077, 32.50000365650003, 598.0000000000028]}), + ('Structured3D', + 'structured3d_00393_room_584/67', + {'category': 'unknown', + 'instance_id': 67, + 'position': {'x': 1.305684, 'y': 0.087636, 'z': 0.4325}, + 'raw_basis': [[-0.9999999999999962, 8.742278110047815e-08, 0.0], + [3.821371129789495e-15, 4.3711388286737763e-08, -0.9999999999999991], + [-8.742278110047807e-08, -0.9999999999999953, -4.3711388175715626e-08]], + 'raw_coeffs': [209.00000000000077, 32.50000365650003, 548.0000000000027]}), + ('Structured3D', + 'structured3d_01159_room_232/100', + {'category': 'table', + 'instance_id': 100, + 'position': {'x': 0.44347, 'y': -2.189658, 'z': 0.3155}, + 'raw_basis': [[4.371139006309477e-08, 3.821370999999997e-15, -0.9999999999999991], + [8.742277999999968e-08, -0.999999999999996, 2.2499312661442353e-22], + [-0.9999999999999953, -8.742277999999959e-08, -4.3711389952072466e-08]], + 'raw_coeffs': [35.00000000000003, 18.75000000000007, 12.452050000000058]}), + ('Structured3D', + 'structured3d_02687_room_670/53', + {'category': 'unknown', + 'instance_id': 53, + 'position': {'x': -0.730234, 'y': -1.066933, 'z': 0.3145}, + 'raw_basis': [[1.0, 0.0, 0.0], + [0.0, -4.371138828673793e-08, -0.9999999999999991], + [-0.0, 0.9999999999999991, -4.3711388175715626e-08]], + 'raw_coeffs': [223.5, 32.50000365650003, 336.00000000000034]}), + ('Structured3D', + 'structured3d_02687_room_670/54', + {'category': 'unknown', + 'instance_id': 54, + 'position': {'x': -0.330234, 'y': -1.066933, 'z': 0.3145}, + 'raw_basis': [[1.0, 0.0, 0.0], + [0.0, -4.371138828673793e-08, -0.9999999999999991], + [-0.0, 0.9999999999999991, -4.3711388175715626e-08]], + 'raw_coeffs': [173.5, 32.50000365650003, 336.00000000000034]}), + ('Structured3D', + 'structured3d_02687_room_670/55', + {'category': 'unknown', + 'instance_id': 55, + 'position': {'x': 0.019766, 'y': -1.066933, 'z': 0.3145}, + 'raw_basis': [[1.0, 0.0, 0.0], + [0.0, -4.371138828673793e-08, -0.9999999999999991], + [-0.0, 0.9999999999999991, -4.3711388175715626e-08]], + 'raw_coeffs': [173.5, 32.50000365650003, 336.00000000000034]}), + ('Structured3D', + 'structured3d_03087_room_332351/56', + {'category': 'bed', + 'instance_id': 56, + 'position': {'x': 1.063394, 'y': 1.34462, 'z': 0.3805}, + 'raw_basis': [[1.9984014443252818e-15, 4.371138999999996e-08, -0.9999999999999991], + [0.9999999999999991, -4.37113899041451e-08, 0.0], + [-4.3711389793122756e-08, -0.9999999999999981, -4.371139e-08]], + 'raw_coeffs': [90.66900000000008, 12.37120000000001, 15.869400000000033]}), + ('Structured3D', + 'structured3d_03155_room_309/16', + {'category': 'table', + 'instance_id': 16, + 'position': {'x': -0.60826, 'y': 4.415584, 'z': 0.2747}, + 'raw_basis': [[-3.3306690738754696e-15, -4.371138999999996e-08, -0.9999999999999991], + [-0.9999999999999972, 7.549789999487077e-08, -1.1102230246251565e-16], + [7.549789999487071e-08, 0.9999999999999962, -4.3711390000000065e-08]], + 'raw_coeffs': [35.00000000000003, 18.750000000000053, 12.452050000000046]}), + ('Structured3D', + 'structured3d_03356_room_584/129', + {'category': 'table', + 'instance_id': 129, + 'position': {'x': 2.124656, 'y': -1.657159, 'z': 0.4325}, + 'raw_basis': [[-4.371139117331779e-08, 0.9999999999999992, 0.0], + [4.3711388286737896e-08, 1.9106855776612683e-15, -0.9999999999999991], + [-0.9999999999999983, -4.3711390840250886e-08, -4.3711388175715626e-08]], + 'raw_coeffs': [209.0000000000002, 32.50000365650003, 273.0000000000005]}), + ('Structured3D', + 'structured3d_03356_room_584/133', + {'category': 'unknown', + 'instance_id': 133, + 'position': {'x': 1.409656, 'y': -0.555107, 'z': 0.4325}, + 'raw_basis': [[-4.371139117331779e-08, -0.9999999999999992, 0.0], + [-4.3711388286737896e-08, 1.9106855776612683e-15, -0.9999999999999991], + [0.9999999999999983, -4.3711390840250886e-08, -4.3711388175715626e-08]], + 'raw_coeffs': [209.0000000000002, 32.50000365650003, 438.00000000000097]}), + ('Structured3D', + 'structured3d_03356_room_584/139', + {'category': 'unknown', + 'instance_id': 139, + 'position': {'x': 1.409656, 'y': -1.657159, 'z': 0.4325}, + 'raw_basis': [[-4.371139117331779e-08, 0.9999999999999992, 0.0], + [4.3711388286737896e-08, 1.9106855776612683e-15, -0.9999999999999991], + [-0.9999999999999983, -4.3711390840250886e-08, -4.3711388175715626e-08]], + 'raw_coeffs': [209.0000000000002, 32.50000365650003, 438.00000000000097]}), + ('HSSD200', + 'HSSD200:102816627:3/102816627:267', + {'category': 'book', + 'scene_id': '102816627', + 'instance_id': '267', + 'obb_center': '[3.546474, 0.121647, -1.923595]', + 'obb_half_extents': '[0.047279, 0.12895, 0.102077]', + 'obb_rotation_wxyz': '[0.65328163, 0.6532815, -0.27059783, -0.27059788]'}), + ('HSSD200', + 'HSSD200:102816627:3/102816627:309', + {'category': 'book', + 'scene_id': '102816627', + 'instance_id': '309', + 'obb_center': '[3.546474, 1.921797, -1.923595]', + 'obb_half_extents': '[0.047279, 0.12895, 0.102077]', + 'obb_rotation_wxyz': '[0.65328163, 0.6532815, -0.27059783, -0.27059788]'}), + ('HSSD200', + 'HSSD200:103997781_171030978:3/103997781_171030978:185', + {'category': 'book', + 'scene_id': '103997781_171030978', + 'instance_id': '185', + 'obb_center': '[-2.296859, 0.115267, -4.154386]', + 'obb_half_extents': '[0.044784, 0.12895, 0.09669]', + 'obb_rotation_wxyz': '[0.65407459, 0.65248737, -0.27139045, -0.26980341]'}), + ('HSSD200', + 'HSSD200:104348082_171512994:0/104348082_171512994:150', + {'category': 'book', + 'scene_id': '104348082_171512994', + 'instance_id': '150', + 'obb_center': '[-9.489737, 0.149516, -1.599131]', + 'obb_half_extents': '[0.0562, 0.104913, 0.121338]', + 'obb_rotation_wxyz': '[0.60044965, 0.31731935, 0.21637841, -0.70139078]'}), + ('HSSD200', + 'HSSD200:104862501_172226556:5/104862501_172226556:165', + {'category': 'book', + 'scene_id': '104862501_172226556', + 'instance_id': '165', + 'obb_center': '[-4.624166, 0.121647, 4.894016]', + 'obb_half_extents': '[0.047279, 0.12895, 0.102077]', + 'obb_rotation_wxyz': '[0.65328142, 0.2705982, 0.27059834, -0.65328136]'}), + ('HSSD200', + 'HSSD200:104862501_172226556:5/104862501_172226556:207', + {'category': 'book', + 'scene_id': '104862501_172226556', + 'instance_id': '207', + 'obb_center': '[-3.159476, 0.121647, -0.588035]', + 'obb_half_extents': '[0.047279, 0.12895, 0.102077]', + 'obb_rotation_wxyz': '[0.65328163, 0.6532815, -0.27059783, -0.27059788]'}), + ('HSSD200', + 'HSSD200:106879005_174887124:4/106879005_174887124:98', + {'category': 'roof', + 'scene_id': '106879005_174887124', + 'instance_id': '98', + 'obb_center': '[-5.67073, 3.956208, -6.830123]', + 'obb_half_extents': '[6.5, 0.55117, 2.749985]', + 'obb_rotation_wxyz': '[0.97029571, -0.24192195, 0.0, 0.0]'}), + ('HSSD200', + 'HSSD200:107734158_175999998:14/107734158_175999998:448', + {'category': 'book', + 'scene_id': '107734158_175999998', + 'instance_id': '448', + 'obb_center': '[-9.930676, 0.157266, -2.633861]', + 'obb_half_extents': '[0.060269, 0.12895, 0.130123]', + 'obb_rotation_wxyz': '[-0.21313372, -0.70393206, 0.59736535, -0.31970036]'}), + ('MultiScan', + 'MultiScan::scene_00029_01/o43', + {'category': 'bag', + 'object_id': '43', + 'obb_center': '[-2.216006052445697, -3.198990954172932, -2.0838447429477776]', + 'obb_axes': '[-0.9391466459853527, 0.34151555401180256, -0.03702301584790649, 0.2533428178007576, ' + '0.6157978529307632, -0.7460632821648807, -0.23199352146824048, -0.7100423442963621, ' + '-0.6648450009610445]', + 'obb_half_extents': '[0.2190989368345001, 0.2549238904394131, 0.1397719718865565]', + 'front': '[0.23199352146824054, 0.7100423442963623, 0.6648450009610446]', + 'is_architectural': 'False', + 'is_opening': 'False'}), + ('MultiScan', + 'MultiScan::scene_00109_04/o31', + {'category': 'backpack', + 'object_id': '31', + 'obb_center': '[0.6333183612225775, 1.7997251504673377, -0.9412570338535274]', + 'obb_axes': '[-0.9961946980917455, -0.022557566113149824, -0.0841859828293692, -0.08715574274765812, ' + '0.25783416049629937, 0.9622501868990583, -2.7755575615628914e-17, 0.9659258262890683, ' + '-0.2588190451025206]', + 'obb_half_extents': '[0.19922853225692852, 0.20651165445598438, 0.11010368672622178]', + 'front': '[2.7755575615628914e-17, -0.9659258262890683, 0.2588190451025206]', + 'is_architectural': 'False', + 'is_opening': 'False'})] + + +def rotation(yaw, roll=0, pitch=0): + c, s = math.cos(yaw), math.sin(yaw) + a, b = math.cos(roll), math.sin(roll) + d, e = math.cos(pitch), math.sin(pitch) + return np.array([[c, -s, 0], [s, c, 0], [0, 0, 1]]) @ np.array( + [[1, 0, 0], [0, a, -b], [0, b, a]]) @ np.array([[d, 0, e], [0, 1, 0], [-e, 0, d]]) + + +def quaternion_matrix(q): + # Independent vector/quaternion formula, applied to the three basis vectors. + w, v = q[0], np.asarray(q[1:]) + return np.column_stack([e + 2 * np.cross(v, w * e + np.cross(v, e)) for e in np.eye(3)]) + + +def corners(R, size, center): + return np.asarray(list(itertools.product((-0.5, 0.5), repeat=3))) * size @ R.T + center + + +def pose(f): + return (np.array([f['furniture_size'][k] for k in ('width', 'length', 'height')]), + np.array([f['furniture_position'][k] for k in 'xyz']), + math.radians(f['furniture_rotation']['z'])) + + +def assert_envelope(f, points, center_z): + size, pos, yaw = pose(f) + local = (points - pos) @ rotation(yaw) + expected_lo = -size / 2 if center_z else np.array([-size[0] / 2, -size[1] / 2, 0]) + np.testing.assert_allclose(local.min(0), expected_lo, atol=1e-8) + np.testing.assert_allclose(local.max(0), expected_lo + size, atol=1e-8) + + +@pytest.mark.parametrize('source,uid,raw', AUDITED_BOXES, ids=[r[1] for r in AUDITED_BOXES]) +def test_all_audited_retained_boxes_enclose_rotated_corners(source, uid, raw): + before = copy.deepcopy(raw) + if source == 'Structured3D': + f = structured3d._furniture(raw) + R, size = np.asarray(raw['raw_basis']).T, 2 * np.abs(raw['raw_coeffs']) / 1000 + center = np.array([raw['position'][k] for k in 'xyz']) + elif source == 'HSSD200': + f = hssd200._furniture(raw, 0) + R = hssd200.T @ quaternion_matrix(json.loads(raw['obb_rotation_wxyz'])) + size = 2 * np.asarray(json.loads(raw['obb_half_extents'])) + center = hssd200.T @ np.asarray(json.loads(raw['obb_center'])) + else: + f = multiscan.box(raw) + R = np.asarray(json.loads(raw['obb_axes'])).reshape(3, 3).T + size = 2 * np.asarray(json.loads(raw['obb_half_extents'])) + center = np.asarray(json.loads(raw['obb_center'])) + assert_envelope(f, corners(R, size, center), source != 'MultiScan') + assert raw == before + + +def source_row(source, R, size, center, category='table', identity='1'): + """Encode one semantic-front X / up Z box in each source's actual axis convention.""" + if source == 'Structured3D': + return {'category': category, 'instance_id': identity, 'position': dict(zip('xyz', center)), + 'raw_basis': [R[:, 1].tolist(), (-R[:, 0]).tolist(), R[:, 2].tolist()], + 'raw_coeffs': (np.asarray(size)[[1, 0, 2]] * 500).tolist()} + C = R[:, [1, 2, 0]] # asset +Z front, +Y up + row = {'category': category, 'obb_half_extents': json.dumps((np.asarray(size)[[1, 2, 0]] / 2).tolist())} + if source == 'HSSD200': + M = hssd200.T.T @ C + w = math.sqrt(1 + np.trace(M)) / 2 + q = [w, (M[2, 1] - M[1, 2]) / (4 * w), (M[0, 2] - M[2, 0]) / (4 * w), + (M[1, 0] - M[0, 1]) / (4 * w)] + return {**row, 'scene_id': 'synthetic', 'instance_id': identity, + 'obb_center': json.dumps((hssd200.T.T @ center).tolist()), 'obb_rotation_wxyz': json.dumps(q)} + return {**row, 'object_id': identity, 'obb_center': json.dumps(list(center)), + 'obb_axes': json.dumps(C.T.flatten().tolist()), 'front': json.dumps(R[:, 0].tolist()), + 'is_architectural': 'False', 'is_opening': 'False'} + + +def furniture(source, row, floor=0): + if source == 'Structured3D': + return structured3d._furniture(row) + if source == 'HSSD200': + return hssd200._furniture(row, floor) + return multiscan.box(row) + + +@pytest.mark.parametrize('source', ['Structured3D', 'HSSD200', 'MultiScan']) +@pytest.mark.parametrize('yaw', [-1.2, 0.7, 2.1]) +def test_tilt_preserves_horizontal_front_and_exact_center(source, yaw): + R, size, center = rotation(yaw, 0.4), np.array([1.4, 0.8, 0.5]), np.array([10., 20., 2.]) + f = furniture(source, source_row(source, R, size, center)) + assert_envelope(f, corners(R, size, center), source != 'MultiScan') + assert math.cos(pose(f)[2] - yaw) == pytest.approx(1) + assert f['furniture_rotation']['x'] != 0 + assert f.get('front_known') is not False + + +@pytest.mark.parametrize('source', ['Structured3D', 'HSSD200', 'MultiScan']) +def test_upright_known_front_unchanged(source): + R, size, center = rotation(0.7), np.array([1.4, 0.8, 0.5]), np.array([10., 20., 2.]) + f = furniture(source, source_row(source, R, size, center)) + assert_envelope(f, corners(R, size, center), source != 'MultiScan') + np.testing.assert_allclose(pose(f)[0], size) + assert pose(f)[2] == pytest.approx(0.7) + assert f['furniture_rotation']['x'] == 0 + + +@pytest.mark.parametrize('source', ['Structured3D', 'MultiScan']) +def test_vertical_front_has_geometry_but_no_claimed_heading(source): + R = rotation(0.7, pitch=math.pi / 2) + size, center = np.array([1.4, 0.8, 0.5]), np.array([10., 20., 2.]) + f = furniture(source, source_row(source, R, size, center)) + assert_envelope(f, corners(R, size, center), source != 'MultiScan') + assert f.get('front_known') is False or f.get('front_vertical') is True + + +ARGS = SimpleNamespace(boundary_types=['polygon', 'hull'], source_anchors={}, anchors=['floor', 'object'], + min_objects=1, max_vertices=0, oob_tol=.1, hidden_max=.3, reject_flagged=[]) + + +def test_hssd_floor_origin_and_support_use_true_envelope(): + R, size, floor = rotation(.7, .4), np.array([1.4, .8, .5]), 7.25 + height = np.ptp(corners(R, size, np.zeros(3))[:, 2]) + table = hssd200._furniture(source_row('HSSD200', R, size, np.array([12., 22., floor + height / 2])), floor) + book = hssd200._furniture(source_row('HSSD200', rotation(.7), np.array([.2, .2, .1]), + np.array([12., 22., floor + height + .05]), 'book', '2'), floor) + room = {'room_id': 'r', 'room_boundary': [[10, 20], [15, 20], [15, 25], [10, 25]], + 'room_height': 3, 'furniture': [table, book]} + ir = convert_room(room, source='HSSD200', uid='r', group='g', boundary_type='polygon', center_z=True, + meta={'front_known': True}) + assert ir['objects'][0]['pos'] == pytest.approx([2, 2, 0]) + built, why = prep(ir, ARGS) + assert why is None + child = next(o for o in built['objects'] if o['id'] == 'synthetic:2') + assert child['anchor'] == 'object' and child['parent'] == 'synthetic:1' + assert child['pos'][2] == round(height, 2) + + +def test_structured_false_floor_obstacle_is_removed_after_preparation(): + raw = AUDITED_BOXES[0][2] # structured3d_00160_room_584 / instance 50 + chair = source_row('Structured3D', rotation(0), np.array([.6, .6, 1.]), + np.array([6., 6., .5]), 'chair', 'control') + record = {'sample_id': 'bounded', 'provenance': {'source_house_id': 'test', 'split': 'train'}, + 'room_origin_house_m': [0, 0, 0], + 'room_context': {'room_id': 'r', 'room_type': 'office', 'ceiling_height_m': 3, + 'floor_polygon': [[0, 0], [10, 0], [10, 10], [0, 10]]}, + 'layout': {'floor_layout': {'objects': [raw, chair]}}} + ir = structured3d.convert(record) + false_obstacle = ir['objects'][0] + assert false_obstacle['pos'][2] == pytest.approx(.4, abs=1e-6) + assert false_obstacle['size'][2] == pytest.approx(.065, abs=1e-6) + assert false_obstacle['front_known'] is False + built, why = prep(ir, ARGS) + assert why is None + assert [o['id'] for o in built['objects']] == ['control'] + assert not built['fixed'] + + +def csv_stream(rows): + stream = io.StringIO() + writer = csv.DictWriter(stream, fieldnames=list(rows[0])) + writer.writeheader() + writer.writerows(rows) + stream.seek(0) + return stream + + +def test_hssd_load_keeps_tilt_envelope_floor_and_front_metadata(): + floor = 7.25 + R, size = rotation(.7, .4), np.array([1.4, .8, .5]) + height = np.ptp(corners(R, size, np.zeros(3))[:, 2]) + center = np.array([12., 22., floor + height / 2]) + raw = source_row('HSSD200', R, size, center) + row = {**raw, 'instance_kind': 'rigid', 'bbox_source': 'mesh', 'region_id': '0', + 'region_source': 'inside', 'is_architectural': 'False', + 'aabb_max': json.dumps((corners(R, size, center) @ hssd200.T).max(0).tolist())} + region = {'scene_id': 'synthetic', 'region_id': '0', 'room_type': 'office', 'region_name': 'test', + 'poly_loop_xz': '[[10,-20],[15,-20],[15,-25],[10,-25]]', + 'floor_height': floor, 'ceiling_height': floor + 3, 'extrusion_height': 3} + files = {'objects.csv': [row], 'regions.csv': [region], + 'asset_catalog.csv': [{'template_name': 'unused', 'local_bbox_min': ''}]} + with patch('builtins.open', side_effect=lambda path: csv_stream(files[path.rsplit('/', 1)[-1]])): + ir, = hssd200.load('unused') + obj, = ir['objects'] + assert obj['pos'] == pytest.approx([2, 2, 0]) + assert obj['yaw'] == pytest.approx(.7) + assert obj['size'][2] == pytest.approx(height) + assert obj['tilted'] and 'rot_wxyz_yup' in obj + assert ir['meta']['floor_height'] == floor + + +def test_multiscan_load_uses_source_floor_and_propagates_unknown_front(): + floor, R, size = -3.25, rotation(.7, pitch=math.pi / 2), np.array([.8, .6, .4]) + row = {**source_row('MultiScan', R, size, np.array([12., 22., floor + .4])), + 'scan_id': 'scene_9_0', 'aabb_min': json.dumps([11, 21, floor])} + files = {'objects.csv': [row], 'relations.csv': [{'scan_id': 'scene_9_0', 'subject_id': '1', + 'relation': 'rests_on_floor'}], + 'scans.csv': [{'scan_id': 'scene_9_0', 'scene_id': 'scene_9', 'room_type': 'office', 'device': 'test'}], + 'regions.csv': [{'scan_id': 'scene_9_0', 'floor_height': floor + .2, 'ceiling_height': floor + 3, + 'height_reliable': 'True', 'poly_source': 'floor', 'wall_segments': '[]', + 'poly_loop': '[[10,20],[15,20],[15,25],[10,25]]'}]} + with patch('builtins.open', side_effect=lambda path: csv_stream(files[path.rsplit('/', 1)[-1]])): + ir, = multiscan.load('unused') + assert ir['height'] == 3 + assert ir['meta']['floor_z'] == floor + assert ir['objects'][0]['pos'] == pytest.approx([2, 2, 0]) + assert ir['objects'][0].get('front_known') is False + + +def interior_box(identity, xy, category='chair'): + pts = [[xy[0] + x, xy[1] + y, z] for z in (0., 1.) + for x, y in ((.2, -.2), (-.2, -.2), (-.2, .2), (.2, .2))] + return {'furniture_category': category, 'furniture_instance_id': identity, + 'furniture_position': {'x': 999, 'y': 999, 'z': 999}, + 'source_fields': {'bounding_box': [dict(zip('xyz', p)) for p in pts]}} + + +def interior_room(identity, ring, furn=()): + return {'room_id': identity, 'room_boundary': ring, 'room_height': 3, 'furniture': list(furn)} + + +def load_interior(scene): + with patch.object(interiorgs, 'iter_records', return_value=iter([scene])): + return list(interiorgs.load('unused')) + + +def test_interior_recovers_only_unique_bbox_centers_without_mutating_assignments(): + # Repeated closing/zero-length edges triggered the upstream ray-cast defect. + left = [[10, 20], [12, 20], [12, 22], [10, 22], [10, 22], [10, 20]] + right = [[12, 20], [14, 20], [14, 22], [12, 22]] + original = interior_box('already', [11, 21]) + candidates = [interior_box('left', [11.5, 21]), interior_box('right', [13, 21]), + interior_box('shared', [12, 21]), interior_box('outside', [100, 100]), + interior_box('already', [13, 21]), + {**interior_box('invalid', [11, 21]), 'source_fields': {'bounding_box': []}}] + scene = {'scene_id': 'arbitrary-house', 'rooms': [interior_room('west', left, [original]), + interior_room('east', right)], 'unassigned_objects': candidates} + before = copy.deepcopy(scene) + rooms = load_interior(scene) + assert [[o['id'] for o in r['objects']] for r in rooms] == [['already', 'left'], ['right']] + assert rooms[0]['objects'][1]['pos'] == pytest.approx([1.5, 1, 0]) + recovery = rooms[0]['meta']['membership_recovery'] + assert recovery['scene_recovered'] == 2 + assert recovery['scene_unresolved'] == {'ambiguous': ['shared'], 'outside': ['outside'], 'invalid_bbox': ['invalid']} + assert recovery['scene_already_assigned'] == ['already'] + assert scene == before + assert all(prep(room, ARGS)[1] is None for room in rooms) + + +def test_interior_does_not_pick_first_overlapping_room_or_nearest_room(): + ring = [[0, 0], [4, 0], [4, 4], [0, 4]] + scene = {'scene_id': 'overlap', 'rooms': [interior_room('a', ring), interior_room('b', ring)], + 'unassigned_objects': [interior_box('both', [1, 1]), interior_box('neither', [5, 5])]} + for r in load_interior(scene): + assert not r['objects'] + assert r['meta']['membership_recovery']['scene_unresolved'] == { + 'ambiguous': ['both'], 'outside': ['neither']} + + +@pytest.mark.parametrize('bad', [None, [], [[0, 0], [1, 1], [1, 0], [0, 1]]]) +def test_interior_invalid_boundary_prevents_unproven_unique_assignment(bad): + scene = {'scene_id': 'bad-ring', 'rooms': [interior_room('valid', [[0, 0], [4, 0], [4, 4], [0, 4]]), + interior_room('invalid', bad)], + 'unassigned_objects': [interior_box('uncertain', [1, 1])]} + for r in load_interior(scene): + assert not r['objects'] + assert r['meta']['membership_recovery']['scene_unresolved'] == {'invalid_boundary': ['uncertain']} + + +def test_interior_duplicate_unassigned_ids_stay_unresolved(): + scene = {'scene_id': 'duplicate', 'rooms': [interior_room('a', [[0, 0], [4, 0], [4, 4], [0, 4]])], + 'unassigned_objects': [interior_box('repeat', [1, 1]), interior_box('repeat', [2, 2])]} + r, = load_interior(scene) + assert not r['objects'] + assert r['meta']['membership_recovery']['scene_unresolved'] == {'duplicate_id': ['repeat', 'repeat']} + + +def test_interior_assigned_only_scene_is_unchanged(): + room = interior_room('a', [[10, 20], [14, 20], [14, 24], [10, 24]], [interior_box('kept', [12, 22])]) + scene = {'scene_id': 'stable', 'rooms': [room]} + ir, = load_interior(scene) + assert ir == load_interior({**scene, 'unassigned_objects': []})[0] + assert 'membership_recovery' not in ir['meta'] + assert ir['objects'][0]['pos'] == pytest.approx([2, 2, 0]) + assert len(ir['objects']) == 1 diff --git a/code/fastfill/tests/test_build_repairs.py b/code/fastfill/tests/test_build_repairs.py new file mode 100644 index 0000000000000000000000000000000000000000..0c65308b8c937f7f143114bea12f24bdb41acf1d --- /dev/null +++ b/code/fastfill/tests/test_build_repairs.py @@ -0,0 +1,211 @@ +"""Build failures must never replace a published dataset.""" +import argparse +import copy +import json +from pathlib import Path +import sys + +import pytest + +from fastfill import build + + +@pytest.fixture +def room(): + return {"uid": "fixture:one", "source": "Fixture", "group": "fixture:one", + "meta": {"front_known": True}, "boundary_type": "polygon", + "boundary": [[0, 0], [5, 0], [5, 5], [0, 5]], "height": 3, + "objects": [{"id": "a", "category": "chair", "size": [1, 1, 1], + "pos": [2, 2, 0], "yaw": 0, "anchor": "floor", "parent": None}]} + + +def invoke(monkeypatch, ir, out, *extra): + monkeypatch.setattr(sys, "argv", ["build", "--ir", str(ir), "--out", str(out), *extra]) + build.main() + + +def inputs(tmp_path, room): + ir = tmp_path / "ir" + ir.mkdir() + (ir / "Fixture.jsonl").write_text(json.dumps(room) + "\n") + return ir + + +@pytest.mark.parametrize("case", ["missing", "empty", "unknown_source", "rejected"]) +def test_bad_input_preserves_existing_dataset(tmp_path, monkeypatch, room, case): + ir = inputs(tmp_path, room) + out = tmp_path / "existing" + out.mkdir() + target = out / "train.jsonl" + target.write_bytes(b"existing training data\n") + extra = [] + if case == "missing": + ir = tmp_path / "missing" + elif case == "empty": + (ir / "Fixture.jsonl").write_text("") + elif case == "unknown_source": + extra = ["--sources", "Absent"] + else: + room["meta"]["front_known"] = False + (ir / "Fixture.jsonl").write_text(json.dumps(room) + "\n") + with pytest.raises((SystemExit, ValueError, FileExistsError)): + invoke(monkeypatch, ir, out, *extra) + assert target.read_bytes() == b"existing training data\n" + assert list(out.iterdir()) == [target] + + +def test_valid_input_does_not_overwrite_existing_dataset(tmp_path, monkeypatch, room): + ir = inputs(tmp_path, room) + out = tmp_path / "existing" + out.mkdir() + (out / "sentinel").write_bytes(b"preserve") + with pytest.raises((SystemExit, ValueError, FileExistsError)): + invoke(monkeypatch, ir, out) + assert (out / "sentinel").read_bytes() == b"preserve" + + +def test_failed_write_leaves_no_partial_output(tmp_path, monkeypatch, room): + ir = inputs(tmp_path, room) + out = tmp_path / "new-version" + def fail(*args, **kwargs): + raise RuntimeError("fixture serialization failure") + monkeypatch.setattr(build, "messages", fail) + with pytest.raises(RuntimeError, match="fixture serialization failure"): + invoke(monkeypatch, ir, out) + assert not out.exists() + assert not list(tmp_path.glob(".new-version-*")) + + +def test_build_writes_complete_manifest_and_does_not_mutate_input(tmp_path, monkeypatch, room): + ir = inputs(tmp_path, room) + before = (ir / "Fixture.jsonl").read_bytes() + out = tmp_path / "new-version" + invoke(monkeypatch, ir, out) + manifest = json.loads((out / "MANIFEST.json").read_text()) + assert sum(manifest["files"][f"{s}.jsonl"]["lines"] for s in ["train", "dev", "test"]) == 1 + assert set(manifest["ir_sha256"]) == {"Fixture.jsonl"} + # Audit/test directories are excluded. A production module such as + # v2/review_data.py is code and must remain fingerprinted. + assert not any(part.startswith("review") or part == "tests" + for p in manifest["code_sha256"] for part in Path(p).parts[:-1]) + assert (ir / "Fixture.jsonl").read_bytes() == before + + +@pytest.mark.parametrize("case", ["empty_selection", "empty_file", "all_rejected", "malformed_json", "invalid_fraction"]) +def test_invalid_build_does_not_create_destination(tmp_path, monkeypatch, room, case): + ir = inputs(tmp_path, room) + out = tmp_path / "new-version" + extra = [] + if case == "empty_selection": + extra = ["--sources"] + elif case == "empty_file": + (ir / "Fixture.jsonl").write_text("") + elif case == "all_rejected": + room["meta"]["front_known"] = False + (ir / "Fixture.jsonl").write_text(json.dumps(room) + "\n") + elif case == "malformed_json": + (ir / "Fixture.jsonl").write_text("{broken\n") + else: + extra = ["--dev", ".8", "--test", ".8"] + with pytest.raises((SystemExit, ValueError)): + invoke(monkeypatch, ir, out, *extra) + assert not out.exists() + + +def test_ir_change_during_build_is_not_published(tmp_path, monkeypatch, room): + ir = inputs(tmp_path, room) + out = tmp_path / "new-version" + original_write = build._write_dataset + def change_input(*args): + original_write(*args) + (ir / "Fixture.jsonl").write_text("changed while building\n") + monkeypatch.setattr(build, "_write_dataset", change_input) + with pytest.raises(RuntimeError, match="IR changed during build"): + invoke(monkeypatch, ir, out) + assert not out.exists() + + +@pytest.mark.parametrize("module,aborts", [("validate.py", True), ("serve.py", False)]) +def test_manifest_distinguishes_build_dependency_edits(tmp_path, monkeypatch, room, module, aborts): + ir = inputs(tmp_path, room) + out = tmp_path / "new-version" + original_sha = build._sha256 + calls = 0 + def edited_hash(path): + nonlocal calls + if str(path).endswith("/" + module): + calls += 1 + if calls > 1: + return "0" * 64 + return original_sha(path) + monkeypatch.setattr(build, "_sha256", edited_hash) + if aborts: + with pytest.raises(RuntimeError, match="implementation changed"): + invoke(monkeypatch, ir, out) + assert not out.exists() + else: + invoke(monkeypatch, ir, out) + manifest = json.loads((out / "MANIFEST.json").read_text()) + assert manifest["code_sha256"][module] == "0" * 64 + assert manifest["build_start_code_sha256"][module] != "0" * 64 + + +@pytest.mark.parametrize("split", ["dev", "test"]) +def test_eval_sidecars_roundtrip_and_constraints_hold(tmp_path, monkeypatch, room, split): + from fastfill.scene import messages + from fastfill.validate import holds + room["objects"] += [{"id": "b", "category": "table", "size": [.8, .8, .7], + "pos": [3, 2, 0], "yaw": 0, "anchor": "floor", "parent": None}] + ir = inputs(tmp_path, room) + out = tmp_path / "new-version" + monkeypatch.setattr(build, "assign_splits", lambda rooms, dev, test: [split] * len(rooms)) + invoke(monkeypatch, ir, out) + row = json.loads((out / f"{split}.jsonl").read_text()) + reference = json.loads((out / f"{split}_rooms.jsonl").read_text()) + constrained = json.loads((out / f"{split}_constrained_rooms.jsonl").read_text()) + assert row["messages"] == messages(reference) + assert constrained["constraints"] + assert all(holds(c, reference["objects"], reference["boundary"]) for c in constrained["constraints"]) + + +def test_training_constraints_survive_augmentation_and_serialization(tmp_path, monkeypatch, room): + from fastfill.scene import parse + from fastfill.validate import holds + room["objects"] += [ + {"id": "b", "category": "table", "size": [.8, .8, .7], "pos": [3, 2, 0], + "yaw": 0, "anchor": "floor", "parent": None}, + {"id": "c", "category": "vase", "size": [.2, .2, .3], "pos": [3, 2, .7], + "yaw": 0, "anchor": "object", "parent": "b"}] + ir = inputs(tmp_path, room) + out = tmp_path / "new-version" + invoke(monkeypatch, ir, out, "--dev", "0", "--test", "0", "--constraint_frac", "1", + "--field_dropout", "0", "--desc_dropout", "0") + row = json.loads((out / "train.jsonl").read_text()) + prompt = json.loads(row["messages"][1]["content"]) + placements, error = parse(row["messages"][2]["content"], [o["id"] for o in prompt["objects"]]) + assert error is None + objects = [{**o, "pos": placements[o["id"]]["pos"], "yaw": placements[o["id"]]["yaw"], + "parent": placements[o["id"]].get("on")} for o in prompt["objects"]] + assert prompt["constraints"] + assert any(c[0] == "on" for c in prompt["constraints"]) + assert all(holds(c, objects, prompt["boundary"]) for c in prompt["constraints"]) + + +@pytest.mark.parametrize("field,value", [ + ("boundary", [[0, 0], [5, 0], [5, float("nan")], [0, 5]]), + ("boundary", [[0, 0], [5, 5]]), + ("boundary", [[0, 0], [0, 0], [0, 0]]), + ("boundary", [[0, 0], [5, 0, 0], [5, 5], [0, 5]]), + ("height", float("inf")), ("height", -2), ("height", "3"), + ("height", 10 ** 400), + ("boundary", [[0, 0], [10 ** 400, 0], [5, 5], [0, 5]]), +]) +def test_malformed_room_geometry_is_rejected(room, field, value): + args = argparse.Namespace(boundary_types=["polygon", "hull"], source_anchors={}, + anchors=["floor", "object"], min_objects=1, max_vertices=0, + oob_tol=.1, hidden_max=.3, reject_flagged=[]) + original = copy.deepcopy(room) + result, reason = build.prep({**room, field: value}, args) + assert result is None + assert reason in {"boundary_shape", "bad_numbers"} + assert room == original diff --git a/code/fastfill/tests/test_embedded_contracts.py b/code/fastfill/tests/test_embedded_contracts.py new file mode 100644 index 0000000000000000000000000000000000000000..2718b3237a653b6fb0b7b6abd26a740a48c7b835 --- /dev/null +++ b/code/fastfill/tests/test_embedded_contracts.py @@ -0,0 +1,18 @@ +"""Keep the existing synthetic geometry and evaluation contract checks in the normal test command.""" +import runpy +import sys +import warnings + +import pytest + + +@pytest.mark.parametrize("module", [ + "fastfill.adapters.structured3d", "fastfill.adapters.hssd200", + "fastfill.adapters.multiscan", "fastfill.adapters.interiorgs", + "fastfill.evaluate", "fastfill.split", "fastfill.anchors", +]) +def test_existing_contracts(module, monkeypatch): + monkeypatch.setattr(sys, "argv", [module]) + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", message=".*found in sys.modules.*", category=RuntimeWarning) + runpy.run_module(module, run_name="__main__") diff --git a/code/fastfill/tests/test_qa_repairs.py b/code/fastfill/tests/test_qa_repairs.py new file mode 100644 index 0000000000000000000000000000000000000000..907d26a3e1d796cc0aebe95807070104419b0995 --- /dev/null +++ b/code/fastfill/tests/test_qa_repairs.py @@ -0,0 +1,454 @@ +"""Tiny, offline regressions for DATA-QA-1/2/3 in the 2026-09-27 audit. + +Run: python -m unittest discover -s fastfill/tests -p test_qa_repairs.py -v +No persisted dataset, trainer dependencies, or model downloads are needed. +""" +import argparse +import ast +import collections +import contextlib +import copy +import hashlib +import io +import json +import math +from pathlib import Path +import subprocess +import sys +import tempfile +import types +import unittest +from unittest.mock import patch + +from fastfill import build +from fastfill.scene import canonical, messages, rot90 +from fastfill.tools import ablation_check, qa_report + + +ROOT = Path(__file__).resolve().parents[1] +EVAL_FILES = ("dev.jsonl", "test.jsonl", "dev_rooms.jsonl", "test_rooms.jsonl", + "dev_constrained_rooms.jsonl", "test_constrained_rooms.jsonl") +ARGS = dict(reject_flagged=[], keep_duplicates=False, min_objects=1, max_vertices=0, + sources=["S"], source_anchors={}, boundary_types=["polygon", "hull"], + anchors=["floor", "object"], oob_tol=.1, hidden_max=.3, + no_rot90=True, constraint_frac=0., field_dropout=0., desc_dropout=0.) + + +def write_rows(path, records): + path.write_text("".join(json.dumps(r, ensure_ascii=False) + "\n" for r in records)) + + +def file_info(path): + raw = path.read_bytes() + return {"sha256": hashlib.sha256(raw).hexdigest(), "lines": len(raw.splitlines())} + + +def source_room(uid="S:train"): + def obj(id_, cat, pos, size, parent=None): + return dict(id=id_, category=cat, size=size, pos=pos, yaw=0, parent=parent, + anchor="object" if parent else "floor", desc="A wooden furnishing beside a wall.") + return dict(uid=uid, source="S", group="world_" + uid, room_type="livingroom", + boundary=[[0, 0], [8, 0], [8, 6], [0, 6]], boundary_type="polygon", height=3, + objects=[obj("a", "chair", [1, 1, 0], [.5, .5, 1]), + obj("b", "chair", [3, 1, 0], [.5, .5, 1]), + obj("m", "stool", [2, 1, 0], [.4, .4, .5]), + obj("t", "table", [5, 4, 0], [1, 1, 1]), + obj("l", "lamp", [5, 4, 1], [.2, .2, .5], "t"), + obj("w", "cabinet", [.25, 3, 0], [.5, 1, 1])], + meta={"front_known": True}) + + +def saved_row(raw, args, split="train"): + """Fixture writer follows the documented build order, independently of QA.""" + room, why = build.prep(copy.deepcopy(raw), argparse.Namespace(**args)) + assert why is None, why + cons, with_desc = None, True + if split == "train": + aug, s2, drop = (build.rng_for(raw["uid"], k) for k in ("aug", "s2", "dropout")) + room = canonical(room if args["no_rot90"] else rot90(room, aug.randrange(4))) + cons = build.extract_constraints(room, s2) if s2.random() < args["constraint_frac"] else None + if drop.random() < args["field_dropout"]: + room = {**room, "room_type": None} + if drop.random() < args["field_dropout"]: + room = {**room, "height": None} + with_desc = drop.random() >= args["desc_dropout"] + else: + room = canonical(room) + return {"uid": raw["uid"], "source": raw["source"], "flags": room["meta"]["flags"], + "messages": messages(room, cons, with_desc=with_desc)}, room + + +def trainer_stub(): + # Audit technique: exercise the real selector on the original QA without importing torch/peft. + tree = ast.parse((ROOT / "train.py").read_text()) + fn = next(n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name == "select") + mod = types.ModuleType("fastfill.train") + mod.collections, mod.hashlib = collections, hashlib + exec(compile(ast.Module(body=[fn], type_ignores=[]), "train.select", "exec"), mod.__dict__) + return mod + + +class QARepairs(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory(prefix="fastfill-qa-test-") + self.addCleanup(self.tmp.cleanup) + self.root = Path(self.tmp.name) + self.ir, self.data = self.root / "ir", self.root / "v3.2" + self.ir.mkdir() + self.data.mkdir() + + def fixture(self, **overrides): + args = {**ARGS, **overrides} + room = source_room() + row, _ = saved_row(room, args) + write_rows(self.ir / "S.jsonl", [room]) + write_rows(self.data / "train.jsonl", [row]) + for name in EVAL_FILES: + write_rows(self.data / name, []) + write_rows(self.data / "stats.json", [{"S": {"scanned": 1, "train_S1": 1}}]) + self.man = {"args": args, "files": {}, "ir_sha256": {"S.jsonl": file_info(self.ir / "S.jsonl")["sha256"]}, + "code_sha256": {name: file_info(ROOT / name)["sha256"] for name in + ("__init__.py", "build.py", "anchors.py", "scene.py", "split.py", "validate.py", "train.py")}} + self.seal() + return row + + def seal(self): + self.man["files"] = {name: file_info(self.data / name) for name in ("train.jsonl", *EVAL_FILES, "stats.json")} + write_rows(self.data / "MANIFEST.json", [self.man]) + + def run_qa(self, options=False): + args = ["qa_report", "--ir", str(self.ir), "--data", str(self.data)] if options else [ + "qa_report", str(self.ir), str(self.data)] + with patch.object(sys, "argv", args), patch.dict(sys.modules, {"fastfill.train": trainer_stub()}), \ + contextlib.redirect_stdout(io.StringIO()): + self.qa_status = qa_report.main() + return json.loads((self.data / "QA.json").read_text()) + + def ablation(self, original, changed, extra=False): + base, cons = self.root / "base", self.root / "cons" + base.mkdir(exist_ok=True) + cons.mkdir(exist_ok=True) + write_rows(base / "train.jsonl", [original]) + write_rows(cons / "train.jsonl", [changed] * (2 if extra else 1)) + for name in EVAL_FILES: + write_rows(base / name, []) + write_rows(cons / name, []) + with contextlib.redirect_stdout(io.StringIO()): + ablation_check.main(str(base), str(cons)) + + def test_ablation_compares_every_metadata_field_and_message(self): + row, _ = saved_row(source_room(), ARGS) + variants = [{**row, "source": "other"}, {**row, "flags": {"oob_objects": 1}}, + {**row, "weight": .5}, {**row, "messages": row["messages"] + [{"role": "user", "content": "extra"}]}] + for field, value in (("role", "assistant"), ("name", "extra")): + variants.append({**row, "messages": [row["messages"][0], {**row["messages"][1], field: value}, row["messages"][2]]}) + for i, changed in enumerate(variants): + with self.subTest(variant=i), self.assertRaises(AssertionError): + self.ablation(row, changed) + + def test_ablation_does_not_equate_boolean_and_numeric_metadata(self): + row, _ = saved_row(source_room(), ARGS) + with self.assertRaises(AssertionError): + self.ablation({**row, "weight": True}, {**row, "weight": 1}) + + def test_ablation_requires_byte_identical_nonempty_eval_files(self): + row, _ = saved_row(source_room(), ARGS) + self.ablation(row, row) + write_rows(self.root / "base" / "dev.jsonl", [row]) + write_rows(self.root / "cons" / "dev.jsonl", [{**row, "source": "other"}]) + with contextlib.redirect_stdout(io.StringIO()), self.assertRaises(AssertionError): + ablation_check.main(str(self.root / "base"), str(self.root / "cons")) + + def test_ablation_allows_only_user_constraints_and_json_whitespace(self): + row, _ = saved_row(source_room(), ARGS) + changed = copy.deepcopy(row) + user = json.loads(changed["messages"][1]["content"]) + changed["messages"][1]["content"] = json.dumps({**user, "constraints": [["near", "chair_1", "chair_2", 2]]}, indent=2) + self.ablation(row, changed) + with self.assertRaises(AssertionError): + self.ablation(row, changed, extra=True) + with self.assertRaises(AssertionError): + self.ablation(changed, changed) # constraints in the base are not the permitted ablation + + def test_train_constraints_check_schema_references_and_saved_answer_truth(self): + row = self.fixture() + user = json.loads(row["messages"][1]["content"]) + cons = [["on", "lamp_1", "table_1"], ["faces", "chair_1", "chair_2"], + ["near", "chair_1", "chair_2", 2], ["against_wall", "cabinet_1"], + ["between", "stool_1", "chair_1", "chair_2"], + ["near", "chair_1", "does_not_exist", .1], ["near", "chair_1", "chair_2", .1], + ["on", "chair_1", "table_1"], ["bogus", "chair_1"], [], ["faces", "chair_1"], + ["near", "chair_1", "chair_2", True], ["near", "chair_1", "chair_2", -1], + ["near", "chair_1", "chair_2", float("nan")]] + row["messages"][1]["content"] = json.dumps({**user, "constraints": cons}) + write_rows(self.data / "train.jsonl", [row]) + q = self.run_qa() + self.assertEqual(q["constraint_reference_validation"]["train"], "5/14") + self.assertEqual(q["constraint_failures"]["train"], {"schema": 6, "reference": 1, "false": 2}) + self.assertEqual(q["object_set_closure"], {"ok": 1}) + + def test_constraint_container_and_unparseable_answer_are_reported(self): + row = self.fixture() + user = json.loads(row["messages"][1]["content"]) + for constraints, answer, reason in (({}, row["messages"][2]["content"], "schema"), + ([["near", "chair_1", "chair_2", 2]], "broken", "answer")): + with self.subTest(reason=reason): + changed = copy.deepcopy(row) + changed["messages"][1]["content"] = json.dumps({**user, "constraints": constraints}) + changed["messages"][2]["content"] = answer + write_rows(self.data / "train.jsonl", [changed]) + q = self.run_qa() + self.assertEqual(q["constraint_failures"]["train"], {reason: 1}) + self.assertEqual(q["constraint_reference_validation"]["train"], "0/1") + + def test_output_hashes_and_line_counts_include_stats_and_missing_files(self): + self.fixture() + (self.data / "dev_constrained_rooms.jsonl").unlink() + (self.data / "stats.json").write_text('{"S":{"scanned":9}}\n\n') + q = self.run_qa() + differences = q["output files differing from MANIFEST (must be empty)"] + self.assertEqual(set(differences), {"dev_constrained_rooms.jsonl", "stats.json"}) + self.assertIn("missing", differences["dev_constrained_rooms.jsonl"]) + self.assertEqual(set(differences["stats.json"]), {"sha256", "lines"}) + + def test_target_corruption_never_reports_reproducible_even_with_updated_hash(self): + row = self.fixture() + answer = json.loads(row["messages"][2]["content"]) + answer["placements"][0]["pos"] = [6, 5, 0] + row["messages"][2]["content"] = json.dumps(answer) + write_rows(self.data / "train.jsonl", [row]) + q = self.run_qa() + self.assertIn("train.jsonl", q["output files differing from MANIFEST (must be empty)"]) + self.assertEqual(q["leakage (all must be 0)"]["rows not reproducible from IR"], 1) + self.seal() + q = self.run_qa() + self.assertEqual(q["output files differing from MANIFEST (must be empty)"], {}) + self.assertEqual(q["SFT reproducibility"]["mismatches"], {"assistant": 1}) + + def test_replay_detects_user_system_role_and_metadata_changes(self): + row = self.fixture() + for component in ("user", "system", "role", "metadata"): + with self.subTest(component=component): + changed = copy.deepcopy(row) + if component == "user": + user = json.loads(changed["messages"][1]["content"]) + changed["messages"][1]["content"] = json.dumps({**user, "room_type": "wrong"}) + elif component == "system": + changed["messages"][0]["content"] = "wrong system" + elif component == "role": + changed["messages"][1]["role"] = "assistant" + else: + changed["flags"] = {"oob_objects": 1} + write_rows(self.data / "train.jsonl", [changed]) + q = self.run_qa() + self.assertEqual(q["leakage (all must be 0)"]["rows not reproducible from IR"], 1) + self.assertEqual(q["SFT reproducibility"]["mismatches"], {"user" if component == "role" else component: 1}) + + def test_manifest_settings_and_named_cli_reproduce_all_splits(self): + self.fixture(no_rot90=False, constraint_frac=1., field_dropout=.5, desc_dropout=1.) + raw = [source_room()] + for split in ("dev", "test"): + room = source_room("S:" + split) + raw.append(room) + row, prepared = saved_row(room, self.man["args"], split) + write_rows(self.data / (split + ".jsonl"), [row]) + write_rows(self.data / (split + "_rooms.jsonl"), [prepared]) + write_rows(self.ir / "S.jsonl", raw) + self.man["ir_sha256"]["S.jsonl"] = file_info(self.ir / "S.jsonl")["sha256"] + self.seal() + q = self.run_qa(options=True) + self.assertEqual(q["rows"], {"train": 1, "dev": 1, "test": 1}) + self.assertEqual(q["SFT reproducibility"]["matched"], 3) + self.assertEqual(q["leakage (all must be 0)"]["rows not reproducible from IR"], 0) + self.assertEqual(q["build"], str(self.data)) + + def test_trainer_drift_is_disclosed_without_suppressing_build_drift(self): + self.fixture() + self.man["code_sha256"].update({"train.py": "old trainer", "scene.py": "old scene"}) + excluded = ["review_old/audit.py", "tests/test_old.py", ".hidden/tool.py"] + self.man["code_sha256"].update(dict.fromkeys(excluded, "old excluded artifact")) + self.seal() + q = self.run_qa() + self.assertEqual(q["build code differing from MANIFEST (must be empty)"], ["scene.py"]) + self.assertIn("train.py", q["other code changed since the build (does not affect the data)"]) + self.assertEqual(q["legacy MANIFEST code entries outside build provenance scope"], sorted(excluded)) + self.assertTrue(set(excluded).isdisjoint(q["other code changed since the build (does not affect the data)"])) + self.assertEqual(q["SFT reproducibility"]["matched"], 1) + + def test_missing_ir_and_prep_rejection_cannot_count_as_reproduced(self): + self.fixture() + for raw, reason in (([], "missing_ir"), ([{**source_room(), "meta": {"front_known": False}}], "prep")): + with self.subTest(reason=reason): + write_rows(self.ir / "S.jsonl", raw) + q = self.run_qa() + self.assertEqual(q["leakage (all must be 0)"]["rows not reproducible from IR"], 1) + self.assertEqual(q["SFT reproducibility"]["mismatches"], {reason: 1}) + + def test_empty_train_reports_empty_percentiles(self): + self.fixture() + write_rows(self.data / "train.jsonl", []) + self.seal() + q = self.run_qa() + self.assertEqual(q["train objects per row"], dict.fromkeys(("p50", "p90", "p99", "max"))) + self.assertEqual(q["SFT reproducibility"]["matched"], 0) + + def test_missing_stats_still_writes_provenance_discrepancy(self): + self.fixture() + (self.data / "stats.json").unlink() + q = self.run_qa() + self.assertEqual(q["output files differing from MANIFEST (must be empty)"]["stats.json"], {"missing": True}) + self.assertIsNone(q["rejections_by_source"]) + + def test_fixed_references_extra_messages_and_false_geometry_remain_failures(self): + row = self.fixture() + user = json.loads(row["messages"][1]["content"]) + user["fixed"] = [{"id": "column_1", "size": [.3, .3, 3], "pos": [7, 5, 0], "yaw": 0}] + user["constraints"] = [["near", "chair_1", "column_1", 100]] + row["messages"][1]["content"] = json.dumps(user) + target = json.loads(row["messages"][2]["content"]) + target["placements"][0]["pos"] = [20, 20, 0] + row["messages"][2]["content"] = json.dumps(target) + row["messages"].append({"role": "assistant", "content": "extra"}) + write_rows(self.data / "train.jsonl", [row]) + q = self.run_qa() + self.assertEqual(q["constraint_failures"]["train"], {"reference": 1}) + self.assertEqual(q["train rows used by default"]["reference_valid"], "0/1") + self.assertEqual(q["train reference inside walls within 1 mm"]["raw"], "0/1") + self.assertEqual(q["SFT reproducibility"]["mismatches"]["extra_messages"], 1) + row["flags"] = {"overlapping_furniture": 1, "fixed_collision": 1, "oob_objects": 0} + write_rows(self.data / "train.jsonl", [row]) + self.assertEqual(self.run_qa()["train rows used by default"]["left_out"], {"flag:fixed_collision": 1}) + + def test_eval_constraint_validation_and_summary_match_existing_metrics(self): + from fastfill.evaluate import score, summarize + self.fixture() + _, room = saved_row(source_room("S:dev"), ARGS, "dev") + write_rows(self.data / "dev_rooms.jsonl", [room]) + write_rows(self.data / "dev_constrained_rooms.jsonl", [ + {**room, "constraints": [["faces", "chair_1", "chair_2"], ["near", "chair_1", "missing", 1], ["unknown"]]}]) + q = self.run_qa() + self.assertEqual(q["constraint_reference_validation"]["dev"], "1/3") + self.assertEqual(q["constraint_failures"]["dev"], {"reference": 1, "schema": 1}) + expected = summarize([score(room, None)])["in_dist"] + actual = q["reference legality raw vs repaired"]["dev/in_dist"] + self.assertEqual(actual, {k: expected[k] for k in actual}) + + def test_actual_builder_roundtrip_with_manifest_settings(self): + write_rows(self.ir / "S.jsonl", [source_room()]) + args = ["build", "--ir", str(self.ir), "--out", str(self.data), "--sources", "S", + "--dev", "0", "--test", "0", "--constraint_frac", "1", "--field_dropout", "1", "--desc_dropout", "1"] + with patch.object(sys, "argv", args), contextlib.redirect_stdout(io.StringIO()): + build.main() + q = self.run_qa(options=True) + self.assertEqual(q["SFT reproducibility"]["matched"], 1) + self.assertEqual(q["SFT reproducibility"]["failed"], 0) + self.assertEqual(q["MANIFEST defaults used"], {}) + self.assertEqual(q["output files differing from MANIFEST (must be empty)"], {}) + self.assertEqual(q["build code differing from MANIFEST (must be empty)"], []) + + def test_data_files_are_streamed_and_hashing_handles_unterminated_last_line(self): + import builtins + row = self.fixture() + (self.data / "train.jsonl").write_text(json.dumps(row)) # No trailing newline. + self.seal() + original_open = builtins.open + + class Bounded: + def __init__(self, file): + self.file = file + + def __enter__(self): + return self + + def __exit__(self, *exc): + self.file.close() + + def __iter__(self): + return iter(self.file) + + def read(self, size=-1): + if not 0 < size <= 1024 * 1024: + raise AssertionError("unbounded dataset read") + return self.file.read(size) + + def readlines(self, *args): + raise AssertionError("materialized dataset") + + def bounded_open(path, mode="r", *args, **kwargs): + file = original_open(path, mode, *args, **kwargs) + return Bounded(file) if str(path).endswith(".jsonl") and "r" in mode else file + + with patch("builtins.open", bounded_open): + q = self.run_qa() + self.ablation(row, row) + self.assertEqual(q["output files differing from MANIFEST (must be empty)"], {}) + self.assertEqual(q["SFT reproducibility"]["matched"], 1) + + def rotated_fixed_row(self, fixed_xy, chair_xy, chair_size): + room = {"boundary": [[0, 0], [4, 0], [4, 4], [0, 4]], "boundary_type": "polygon", "height": 3, + "fixed": [{"id": "column_1", "category": "column", "size": [2, .1, 2], + "pos": [*fixed_xy, 0], "yaw": math.pi / 2}], + "objects": [{"id": "chair_1", "category": "chair", "size": chair_size, + "pos": [*chair_xy, 0], "yaw": 0, "parent": None}]} + return {"uid": "S:rotated", "source": "S", "flags": {}, "messages": messages(room)} + + def test_rotated_fixed_yaw_is_radians_for_collision_geometry(self): + row = self.rotated_fixed_row([2, 2], [2, 2.8], [.2, .2, 1]) + original = copy.deepcopy(row) + user, room, _, layout, error = qa_report._layout(row) + self.assertIsNone(error) + # A vertical 2m column crosses this chair; 90 radians would miss it. + self.assertEqual(qa_report.check(layout)["fixed_blocking"], [("chair_1", "column_1")]) + self.assertAlmostEqual(room["fixed"][0]["yaw"], math.pi / 2) + self.assertEqual(user["fixed"][0]["yaw"], 90) + self.assertEqual(row, original) + + def test_rotated_fixed_blocks_repair_and_changes_reported_containment(self): + row = self.rotated_fixed_row([.5, 2.8], [.2, 2], [.5, .5, 1]) + # The chair starts 5cm outside the left wall, touching the column's x=.45 edge. + # Every allowed move inside would increase overlap with the vertical column. + q = qa_report._train_metrics(iter([row])) + self.assertEqual(q["train reference inside walls within 1 mm"], {"raw": "0/1", "repaired": "0/1"}) + + def test_mandatory_failures_exclude_trainer_drift_and_geometry_statistics(self): + self.fixture() + self.man["code_sha256"]["train.py"] = "old trainer" + self.seal() + q = self.run_qa() + self.assertEqual(self.qa_status, 0) + self.assertEqual(q["mandatory_failures"], []) + informational = {**q, "train reference inside walls within 1 mm": {"raw": "0/1", "repaired": "0/1"}, + "train rows used by default": {"rows": 1, "reference_valid": "0/1"}} + self.assertEqual(qa_report._mandatory_failures(informational), []) + mandatory = { + "build code differing from MANIFEST (must be empty)": ["scene.py"], + "IR files differing from MANIFEST (must be empty)": ["S.jsonl"], + "output files differing from MANIFEST (must be empty)": {"train.jsonl": {"missing": True}}, + "output files missing from MANIFEST (must be empty)": ["stats.json"], + "object_set_closure": {"ok": 1, "missing_ids": 1}, + "fixed_boxes_answered (must be 0)": 1, + "constraint_failures": {"train": {"false": 1}, "dev": {}, "test": {}}, + "leakage (all must be 0)": {"train&dev worlds": 1}, + "SFT reproducibility": {"matched": 0, "failed": 1}, + } + for field, value in mandatory.items(): + with self.subTest(field=field): + self.assertIn(field, qa_report._mandatory_failures({**q, field: value})) + + def test_cli_writes_report_before_failing_for_target_corruption(self): + row = self.fixture() + command = [sys.executable, "-B", "-m", "fastfill.tools.qa_report", "--ir", str(self.ir), "--data", str(self.data)] + clean = subprocess.run(command, cwd=ROOT.parent, capture_output=True, text=True) + self.assertEqual(clean.returncode, 0, clean.stderr) + row["messages"][2]["content"] = row["messages"][2]["content"].replace('"pos":[5,4,0]', '"pos":[6,5,0]') + write_rows(self.data / "train.jsonl", [row]) + changed = subprocess.run(command, cwd=ROOT.parent, capture_output=True, text=True) + self.assertEqual(changed.returncode, 1, changed.stderr) + q = json.loads((self.data / "QA.json").read_text()) + self.assertIn("output files differing from MANIFEST (must be empty)", q["mandatory_failures"]) + self.assertIn("SFT reproducibility", q["mandatory_failures"]) + self.assertEqual(json.loads(changed.stdout)["mandatory_failures"], q["mandatory_failures"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/code/fastfill/tests/test_runtime_repairs.py b/code/fastfill/tests/test_runtime_repairs.py new file mode 100644 index 0000000000000000000000000000000000000000..89c54de8ad8bb289aec09abd768650dabeb854cd --- /dev/null +++ b/code/fastfill/tests/test_runtime_repairs.py @@ -0,0 +1,388 @@ +"""Bounded runtime audit regressions; synthetic layouts and handler sinks, never sockets.""" +import ast +import copy +import io +import json +import math +from pathlib import Path +import runpy +import unittest +from unittest.mock import Mock +import warnings + +from shapely.geometry import Polygon + +from fastfill import serve, validate +from fastfill.interface import placements_from_text, request_to_room +from fastfill.scene import messages + + +def request(): + return { + "room": {"boundary_xy": [[0, 0], [4, 0], [4, 4], [0, 4]]}, + "objects_to_place": [ + {"id": "a", "category": "box", "size_xyz_m": [.5, .5, .3]}, + {"id": "b", "category": "table", "size_xyz_m": [1, 1, .7]}, + ], + } + + +def model_text(supported=True): + box = {"id": "box_1", "pos": [2, 2, .7] if supported else [1, 1, 0], "yaw": 0} + return json.dumps({"placements": [ + {"id": "table_1", "pos": [2, 2, 0], "yaw": 0}, + {**box, **({"on": "table_1"} if supported else {})}, + ]}) + + +def one_box_text(x, y=2): + return json.dumps({"placements": [{"id": "box_1", "pos": [x, y, 0], "yaw": 0}]}) + + +class RequestRepairs(unittest.TestCase): + def test_anchors_are_enforced_and_reported_without_new_model_protocol(self): + for anchor in ("floor", "object"): + for supported in (False, True): + with self.subTest(anchor=anchor, supported=supported): + req = request() + req["objects_to_place"][0]["anchor"] = anchor + original = copy.deepcopy(req) + generate = Mock(return_value=model_text(supported)) + status, out = serve.answer(req, generate) + expected = supported == (anchor == "object") + self.assertEqual(status, 200 if expected else 422) + validation = (out if status == 200 else out["error"])["validation"] + for phase in ("raw", "repaired"): + report = next(c for c in validation[phase]["constraints"] + if c["constraint"] == ["anchor", "a", anchor]) + self.assertEqual(report["holds"], expected) + self.assertTrue(report["hard"]) + self.assertFalse(report["model_conditioned"]) + self.assertFalse(report["trained"]) + prompt = json.loads(generate.call_args.args[0][1]["content"]) + self.assertNotIn("constraints", prompt) + self.assertTrue(all("anchor" not in o for o in prompt["objects"])) + self.assertEqual(req, original) + + def test_object_anchor_still_requires_geometric_support(self): + req = request() + req["objects_to_place"][0]["anchor"] = "object" + status, out = serve.answer(req, lambda _: model_text().replace('[2, 2, 0.7]', '[1, 1, 0.7]')) + self.assertEqual(status, 422) + self.assertEqual(out["error"]["validation"]["repaired"]["support_fail"], ["a"]) + + def test_soft_keepout_preserves_floor_and_reports_satisfaction(self): + polygon = [[0, 0], [1, 0], [1, 4], [0, 4]] + for x, expected in ((.5, False), (1.25, True), (2, True)): + with self.subTest(x=x): + req = {**request(), "objects_to_place": request()["objects_to_place"][:1], + "layout_constraints": [{"type": "keepout", "hard": False, "polygon_xy": polygon}]} + room, ctx = request_to_room(req) + self.assertEqual(room["boundary"], req["room"]["boundary_xy"]) + self.assertEqual(ctx["floor"].area, 16) + self.assertEqual((ctx["x0"], ctx["y0"]), (0, 0)) + self.assertEqual(room["constraints"], []) + status, out = serve.answer(req, lambda _: one_box_text(x)) + self.assertEqual(status, 200) + self.assertEqual(out["placements"][0]["position_m"], [x, 2, 0]) + for phase in ("raw", "repaired"): + v = out["validation"][phase] + self.assertTrue(v["ok"]) + report = next(c for c in v["constraints"] if c["constraint"] == ["keepout", polygon]) + self.assertEqual(report["holds"], expected) + self.assertFalse(report["hard"]) + self.assertFalse(report["model_conditioned"]) + + def test_full_room_soft_keepout_is_not_rejected_before_generation(self): + req = request() + req["layout_constraints"] = [{"type": "keepout", "hard": False, + "polygon_xy": req["room"]["boundary_xy"]}] + status, out = serve.answer(req, lambda _: model_text()) + self.assertEqual(status, 200) + self.assertFalse(out["validation"]["repaired"]["constraints"][0]["holds"]) + + def test_mixed_keepouts_report_in_request_frame_and_only_cut_hard_geometry(self): + req = {**request(), "room": {"floor_z": 3, + "boundary_xy": [[10, 20], [14, 20], [14, 24], [10, 24]]}, + "objects_to_place": request()["objects_to_place"][:1]} + hard = {"type": "keepout", "polygon_xy": [[10, 20], [11, 20], [11, 24], [10, 24]]} + soft = {"type": "keepout", "hard": False, + "polygon_xy": [[12, 20], [13, 20], [13, 24], [12, 24]]} + req["layout_constraints"] = [hard, soft] + room, ctx = request_to_room(req) + hard_room, hard_ctx = request_to_room({**req, "layout_constraints": [hard]}) + self.assertEqual(room["boundary"], hard_room["boundary"]) + self.assertTrue(ctx["floor"].equals(hard_ctx["floor"])) + out = placements_from_text(one_box_text(12.5 - ctx["x0"]), room, ctx) + self.assertEqual(out["placements"][0]["position_m"], [12.5, 22, 3]) + self.assertTrue(out["validation"]["repaired"]["ok"]) + report = next(c for c in out["validation"]["repaired"]["constraints"] if not c["hard"]) + self.assertEqual(report["constraint"], ["keepout", soft["polygon_xy"]]) + self.assertFalse(report["holds"]) + + def test_hard_keepout_still_excludes_floor(self): + req = {**request(), "objects_to_place": request()["objects_to_place"][:1], + "layout_constraints": [{"type": "keepout", "polygon_xy": [[0, 0], [1, 0], [1, 4], [0, 4]]}]} + _, ctx = request_to_room(req) + self.assertAlmostEqual(ctx["floor"].area, 12) + status, out = serve.answer(req, lambda _: one_box_text(.5 - ctx["x0"])) + self.assertEqual(status, 422) + self.assertEqual(out["error"]["validation"]["repaired"]["outside_1mm"], ["a"]) + + def test_keepout_hard_flag_and_polygon_are_validated(self): + for hard in ("false", 0, None, []): + with self.subTest(hard=hard): + req = {**request(), "layout_constraints": [{"type": "keepout", "hard": hard, + "polygon_xy": [[0, 0], [1, 0], [1, 4], [0, 4]]}]} + with self.assertRaises(ValueError): + request_to_room(req) + for polygon in ([], [[0, 0], [1, 0], [2, 0]], [[0, 0], [2, 2], [0, 2], [2, 0]]): + for hard in (False, True): + with self.subTest(polygon=polygon, hard=hard): + with self.assertRaises(ValueError): + request_to_room({**request(), "layout_constraints": [ + {"type": "keepout", "hard": hard, "polygon_xy": polygon}]}) + + def test_floor_id_must_be_a_string_before_generation(self): + for floor_id in (["invalid"], {}, None, 1, True): + with self.subTest(floor_id=floor_id): + req = request() + req["room"]["floor_id"] = floor_id + generate = Mock(return_value=model_text()) + status, out = serve.answer(req, generate) + self.assertEqual(status, 422) + self.assertEqual(out["error"]["type"], "invalid_request_error") + generate.assert_not_called() + req = request() + req["room"]["floor_id"] = "room-floor" + status, out = serve.answer(req, lambda _: model_text()) + self.assertEqual(status, 200) + self.assertEqual(out["placements"][1]["support_parent"], "room-floor") + + def test_windows_singular_and_plural_have_the_same_collision_policy(self): + results = [] + for category in ("window", "windows", "sliding windows"): + req = request() + req["room"]["fixed_geometry"] = [{"id": "fx", "category": category, + "size_xyz_m": [1, .1, 1.2], "position_m": [2, 2, .4]}] + status, out = serve.answer(req, lambda _: model_text()) + self.assertEqual(status, 200) + results.append(out["validation"]["repaired"]) + self.assertTrue(all(v["fixed_collisions"] == [["b", "fx"], ["a", "fx"]] for v in results)) + self.assertTrue(all(v["fixed_blocking"] == [] for v in results)) + + def test_structural_wall_band_matches_training_and_respects_explicit_kind(self): + from fastfill.build import _keep_fixed + + cases = [("door", None, True), ("windows", None, True), ("cabinet", None, False), + ("gate", "structure", True), ("door", "generic", False)] + for category, kind, kept in cases: + for x in (-.18, -.4): + with self.subTest(category=category, kind=kind, x=x): + fx = {"id": "fx", "category": category, "size_xyz_m": [.07, 1.13, 2.21], + "position_m": [x, .67, 0], "yaw_rad": math.pi, + **({"fixed_kind": kind} if kind else {})} + req = request() + req["room"]["fixed_geometry"] = [fx] + room, ctx = request_to_room(req) + expected = kept and x == -.18 + self.assertEqual(bool(room["fixed"]), expected) + training = _keep_fixed([{"id": "fx", "category": category, "size": fx["size_xyz_m"], + "pos": fx["position_m"], "yaw": math.pi, + "fixed_kind": kind or ("structure" if kept else "generic")}], [], + Polygon(req["room"]["boundary_xy"])) + self.assertEqual(len(room["fixed"]), len(training)) + self.assertEqual(ctx["unsupported"], [] if expected else [{"id": "fx", "reason": "fixed_ignored"}]) + + +class SupportTolerance(unittest.TestCase): + def test_audited_raw_and_rounded_support_examples(self): + # Minimal captured geometry only: portable regression fixtures, without audit-directory dependencies. + cases = [{'label': 'layout_d0d0e745:geometry', + 'parent': {'category': 'shelf', + 'size': [0.24, 0.68, 0.99], + 'pos': [0.19, 3.85, 0.0], + 'yaw': 0.0, + 'id': 'p', + 'parent': None}, + 'child': {'category': 'bottle', + 'size': [0.18, 0.26, 0.2], + 'pos': [0.2, 3.46, 0.99], + 'yaw': 6.178465552059927, + 'id': 'c', + 'parent': 'p'}}, + {'label': 'layout_d0d0e745:raw_geometry', + 'parent': {'category': 'shelf', + 'size': [0.23705480407808122, 0.6822268177839373, 0.9850224800665455], + 'pos': [3.85, 7.813972597960959, 0.0], + 'yaw': 4.71238898038469, + 'id': 'p', + 'parent': None}, + 'child': {'category': 'bottle1', + 'size': [0.18238561321686464, 0.2597533206243067, 0.1978236176908561], + 'pos': [3.4638687850577643, 7.798155180754215, 0.9749873957139794], + 'yaw': 4.607920657487361, + 'id': 'c', + 'parent': 'p'}}, + {'label': 'layout_b4156653:geometry', + 'parent': {'category': 'bookshelf', + 'size': [0.22, 0.65, 1.02], + 'pos': [0.18, 4.45, 0.0], + 'yaw': 0.0, + 'id': 'p', + 'parent': None}, + 'child': {'category': 'bookshelfbook', + 'size': [0.15, 0.32, 0.32], + 'pos': [0.34, 4.36, 1.02], + 'yaw': 1.3788101090755203, + 'id': 'c', + 'parent': 'p'}}, + {'label': 'layout_b4156653:raw_geometry', + 'parent': {'category': 'bookshelf', + 'size': [0.22219992770050281, 0.6485300549974972, 1.0189980105172654], + 'pos': [4.45, 3.821400036149748, 0.0], + 'yaw': 4.71238898038469, + 'id': 'p', + 'parent': None}, + 'child': {'category': 'bookshelfbook_5', + 'size': [0.15457270169708442, 0.31913111738012023, 0.32005607231815225], + 'pos': [4.356301125825175, 3.6611010282046643, 0.9947451596630095], + 'yaw': 6.091614619437346, + 'id': 'c', + 'parent': 'p'}}] + for case in cases: + with self.subTest(case=case['label']): + child, parent = case['child'], case['parent'] + self.assertEqual(validate.check({"objects": [parent, child]})["support_fail"], []) + self.assertTrue(validate.support_ok(child, parent)) + + def test_inclusive_xy_height_and_floor_tolerances_with_epsilon(self): + parent = {"id": "p", "category": "table", "pos": [2, 2, 0], "size": [1, 1, 1], "yaw": 0} + for delta, expected in ((0, True), (5e-10, True), (1e-8, False), (1e-5, False)): + for axis in ("xy", "top", "bottom", "floor", "sunk"): + with self.subTest(delta=delta, axis=axis): + offset = .05 + delta + pos = [2.5 + offset, 2, 1] if axis == "xy" else [2, 2, 1 + offset] if axis == "top" \ + else [2, 2, 1 - offset] if axis == "bottom" else [2, 2, offset if axis == "floor" else -offset] + child = {"id": "c", "category": "box", "size": [.1, .1, .1], "pos": pos, "yaw": 0, + "parent": "p" if axis in ("xy", "top", "bottom") else None} + failures = validate.check({"objects": [parent, child]})["support_fail"] + self.assertEqual(failures, [] if expected else ["c"]) + self.assertEqual(validate.support_ok(child, parent if child["parent"] else None), expected) + + def test_xy_tolerance_is_euclidean_including_rounded_buffer_corners(self): + parent = {"pos": [0, 0, 0], "size": [1, 1, 1], "yaw": 0} + for distance, expected in ((.05, True), (.050001, False)): + angle = math.pi / 64 # between Shapely buffer vertices + xy = [.5 + distance * math.cos(angle), .5 + distance * math.sin(angle)] + self.assertEqual(validate.support_contains(parent, xy), expected) + + def test_missing_parent_still_fails(self): + child = {"id": "c", "category": "box", "size": [.1, .1, .1], "pos": [1, 1, 0], + "yaw": 0, "parent": "missing"} + self.assertEqual(validate.check({"objects": [child]})["support_fail"], ["c"]) + + def test_repair_uses_the_same_inclusive_support_boundary(self): + req = request() + req["objects_to_place"][0]["size_xyz_m"] = [.1, .1, .2] + room, ctx = request_to_room(req) + text = json.dumps({"placements": [ + {"id": "table_1", "pos": [3.4, 2, 0], "yaw": 0}, + {"id": "box_1", "pos": [4, 2, .7], "yaw": 0, "on": "table_1"}]}) + out = placements_from_text(text, room, ctx) + self.assertTrue(out["validation"]["repaired"]["ok"]) + self.assertAlmostEqual(out["placements"][0]["position_m"][0], 3.95) + + +class HttpRepairs(unittest.TestCase): + def dispatch(self, data, length=None, path="/v1/chat/completions"): + generate = Mock(return_value=model_text()) + handler = serve.make_handler(generate, "ff-test", "/ckpt/ff-test").__new__( + serve.make_handler(generate, "ff-test", "/ckpt/ff-test")) + handler.path = path + handler.headers = {} if length is None else {"Content-Length": length} + handler.rfile = Mock(wraps=io.BytesIO(data)) + handler.wfile = io.BytesIO() + handler.send_response = Mock() + handler.send_header = Mock() + handler.end_headers = Mock() + handler.do_POST() + return handler, generate, handler.send_response.call_args.args[0], json.loads(handler.wfile.getvalue()) + + def body(self, content=None): + return json.dumps({"messages": [{"role": "user", "content": json.dumps(request()) if content is None else content}]}).encode() + + def test_invalid_content_lengths_are_rejected_before_read(self): + for length in (None, "", "-1", "0", "abc", "1.5"): + with self.subTest(length=length): + handler, generate, status, out = self.dispatch(self.body(), length) + self.assertEqual(status, 400) + handler.rfile.read.assert_not_called() + generate.assert_not_called() + self.assertEqual(out["error"]["type"], "invalid_request_error") + self.assertEqual(out["error"]["model"], "ff-test") + self.assertEqual(out["error"]["checkpoint"], "/ckpt/ff-test") + + def test_maximum_content_length_is_inclusive_and_oversize_is_413(self): + limit = getattr(serve, "MAX_REQUEST_BYTES", 1024 * 1024) + handler, generate, status, out = self.dispatch(b"", str(limit + 1)) + self.assertEqual(status, 413) + handler.rfile.read.assert_not_called() + generate.assert_not_called() + self.assertEqual(out["error"]["type"], "invalid_request_error") + self.assertEqual(out["error"]["checkpoint"], "/ckpt/ff-test") + body = self.body() + padded = body + b" " * (limit - len(body)) + handler, _, status, _ = self.dispatch(padded, str(limit)) + self.assertEqual(status, 200) + handler.rfile.read.assert_called_once_with(limit) + + def test_deep_outer_and_inner_json_receive_structured_400(self): + nested = "[" * 10000 + "]" * 10000 + for body in (nested.encode(), self.body(nested)): + with self.subTest(inner=body.startswith(b"{")): + _, generate, status, out = self.dispatch(body, str(len(body))) + self.assertEqual(status, 400) + generate.assert_not_called() + self.assertEqual(out["error"]["type"], "invalid_request_error") + self.assertEqual(out["error"]["model"], "ff-test") + + def test_normal_http_response_and_errors_keep_the_existing_envelope(self): + for content, expected in ((None, 200), ("null", 422), ("{bad json", 400)): + body = self.body(content) + _, generate, status, out = self.dispatch(body, str(len(body))) + self.assertEqual(status, expected) + if status == 200: + generate.assert_called_once() + result = json.loads(out["choices"][0]["message"]["content"]) + self.assertEqual(result["checkpoint"], "/ckpt/ff-test") + self.assertEqual(result["model"], "ff-test") + self.assertTrue(result["validation"]["repaired"]["ok"]) + else: + self.assertEqual(out["error"]["model"], "ff-test") + + +class EmbeddedChecks(unittest.TestCase): + def test_scene_validate_interface_selfchecks(self): + with warnings.catch_warnings(): + warnings.simplefilter("ignore", RuntimeWarning) + for name in ("scene", "validate", "interface"): + with self.subTest(module=name): + runpy.run_module("fastfill." + name, run_name="__main__") + + def test_serve_answer_selfchecks_without_binding(self): + root = ast.parse(Path(serve.__file__).read_text()) + main = next(n for n in root.body if isinstance(n, ast.If) + and ast.unparse(n.test) == "__name__ == '__main__'") + nodes = [] + for node in main.body[0].orelse: + if any(isinstance(n, ast.Call) and isinstance(n.func, ast.Name) and n.func.id == "HTTPServer" + for n in ast.walk(node)): + break + nodes.append(node) + self.assertTrue(any(isinstance(n, ast.Assert) for n in nodes)) + exec(compile(ast.Module(body=nodes, type_ignores=[]), serve.__file__, "exec"), dict(vars(serve))) + + +if __name__ == "__main__": + unittest.main() diff --git a/code/fastfill/tests/test_training_repairs.py b/code/fastfill/tests/test_training_repairs.py new file mode 100644 index 0000000000000000000000000000000000000000..44c826b9984144ea51b9af05ac6008a8f4462081 --- /dev/null +++ b/code/fastfill/tests/test_training_repairs.py @@ -0,0 +1,283 @@ +"""Bounded offline regressions for evaluation, merge artifacts, and download exits.""" +import importlib.util +import json +import runpy +import socket +import sys +import types +from pathlib import Path +from unittest.mock import Mock + +import numpy as np +import pytest + +ROOT = Path(__file__).resolve().parents[2] + + +@pytest.fixture(autouse=True) +def offline(monkeypatch): + monkeypatch.setenv("HF_HUB_OFFLINE", "1") + monkeypatch.setenv("TRANSFORMERS_OFFLINE", "1") + + def denied(*args, **kwargs): + raise AssertionError("These tests must not access the network") + + monkeypatch.setattr(socket.socket, "connect", denied) + + +def load_merge(): + spec = importlib.util.spec_from_file_location("merge_under_test", ROOT / "fastfill/merge_lora.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +@pytest.fixture +def hf(monkeypatch): + calls = [] + + class Batch(dict): + def to(self, device): + return self + + class Tokenizer: + @classmethod + def from_pretrained(cls, *args, **kwargs): + return cls() + + def __call__(self, prompts, **kwargs): + assert kwargs.get("add_special_tokens") is False + if isinstance(prompts, str): + return {"input_ids": [ord(c) for c in prompts]} + width = max(map(len, prompts)) + ids = np.array([[0] * (width - len(p)) + [ord(c) for c in p] for p in prompts]) + return Batch(input_ids=ids, attention_mask=(ids != 0).astype(int)) + + def batch_decode(self, rows, **kwargs): + return ["".join(chr(int(c)) for c in row) for row in rows] + + class Model: + device = "cpu" + + @classmethod + def from_pretrained(cls, *args, **kwargs): + return cls() + + def generate(self, input_ids, attention_mask, max_new_tokens, **kwargs): + calls.append((attention_mask.sum(axis=1).tolist(), max_new_tokens)) + assert kwargs["do_sample"] is False + assert input_ids.shape[1] + max_new_tokens == 10 + return np.concatenate((input_ids, np.repeat(input_ids[:, -1:], max_new_tokens, axis=1)), axis=1) + + monkeypatch.setitem(sys.modules, "vllm", None) + monkeypatch.setitem(sys.modules, "torch", types.SimpleNamespace(bfloat16="bfloat16")) + monkeypatch.setitem(sys.modules, "transformers", types.SimpleNamespace( + AutoTokenizer=Tokenizer, AutoModelForCausalLM=Model)) + return calls + + +@pytest.mark.parametrize("batch", [1, 2, 3, 20]) +def test_hf_preserves_each_prompt_budget_and_order(hf, batch): + from fastfill.evaluate import generate + + prompts = ["a", "bbbbbbbb", "cc", "d", "e" * 10, "f" * 11, "gg"] + assert generate("mock", prompts, 10, batch) == ["a" * 9, "bb", "c" * 8, "d" * 9, None, None, "g" * 8] + assert all(len(lengths) <= batch for lengths, _ in hf) + if batch > 1: + assert any(len(lengths) == 2 for lengths, _ in hf) + + +@pytest.mark.parametrize("prompts", [[], ["x" * 10, "y" * 11]]) +def test_hf_does_not_generate_over_limit_prompts(hf, prompts): + from fastfill.evaluate import generate + + assert generate("mock", prompts, 10, 2) == [None] * len(prompts) + assert hf == [] + + +@pytest.fixture +def merge_mock(monkeypatch): + class Base: + def save_pretrained(self, path): + Path(path).mkdir(parents=True, exist_ok=True) + (Path(path) / "config.json").write_text(json.dumps({"rope_parameters": {"rope_theta": 1000000}})) + + class Tokenizer: + def __init__(self, source): + self.config = json.loads((Path(source) / "tokenizer_config.json").read_text()) + + def save_pretrained(self, path): + (Path(path) / "tokenizer_config.json").write_text(json.dumps(self.config)) + + loader = Mock(side_effect=lambda path, **kwargs: Tokenizer(path)) + monkeypatch.setitem(sys.modules, "torch", types.SimpleNamespace(bfloat16="bfloat16")) + monkeypatch.setitem(sys.modules, "transformers", types.SimpleNamespace( + AutoTokenizer=types.SimpleNamespace(from_pretrained=loader), + AutoModelForCausalLM=types.SimpleNamespace(from_pretrained=Mock(return_value=Base())))) + monkeypatch.setitem(sys.modules, "peft", types.SimpleNamespace( + PeftModel=types.SimpleNamespace(from_pretrained=Mock(return_value=types.SimpleNamespace( + merge_and_unload=lambda: Base()))))) + return load_merge(), loader + + +def merge_paths(tmp_path): + base, adapter, merged = tmp_path / "base", tmp_path / "run/final", tmp_path / "merged" + base.mkdir() + adapter.mkdir(parents=True) + (base / "tokenizer_config.json").write_text(json.dumps({"pad_token": None, "extra_special_tokens": ["base"]})) + return base, adapter, merged + + +def test_merge_preserves_adapter_tokenizer_and_compatibility(merge_mock, tmp_path): + merge, loader = merge_mock + base, adapter, merged = merge_paths(tmp_path) + (adapter / "tokenizer_config.json").write_text(json.dumps({ + "pad_token": "", "chat_template": "adapter template", "extra_special_tokens": ["trained"]})) + merge.apply_lora(str(base), str(merged), str(adapter)) + tc = json.loads((merged / "tokenizer_config.json").read_text()) + assert tc == {"pad_token": "", "chat_template": "adapter template", "additional_special_tokens": ["trained"]} + assert Path(loader.call_args.args[0]) == adapter + cfg = json.loads((merged / "config.json").read_text()) + assert cfg["rope_theta"] == cfg["rope_parameters"]["rope_theta"] == 1000000 + + +def test_merge_falls_back_only_when_adapter_tokenizer_absent(merge_mock, tmp_path): + merge, loader = merge_mock + base, adapter, merged = merge_paths(tmp_path) + merge.apply_lora(str(base), str(merged), str(adapter)) + assert Path(loader.call_args.args[0]) == base + (adapter / "tokenizer_config.json").write_text("invalid json") + with pytest.raises(json.JSONDecodeError): + merge.apply_lora(str(base), str(tmp_path / "bad"), str(adapter)) + assert not (tmp_path / "bad").exists() + + +@pytest.mark.parametrize("local", [False, True]) +def test_merge_preserves_one_manifest_family(merge_mock, tmp_path, local): + merge, _ = merge_mock + base, adapter, merged = merge_paths(tmp_path) + run = adapter if local else adapter.parent + manifests = {"run_manifest.json": '{"data":"v3"}\n', + "run_manifest.resume1.json": '{"data":"v3.1"}\n', + "run_manifest.resume10.json": '{"args":{"resume":true}}\n'} + for name, content in manifests.items(): + (run / name).write_text(content) + (run / "run_manifest.resume-backup.json").write_text("not a run") + if local: + (adapter.parent / "run_manifest.json").write_text('{"unrelated":true}') + (adapter.parent / "run_manifest.resume2.json").write_text('{"unrelated":true}') + merge.apply_lora(str(base), str(merged), str(adapter)) + actual = {p.name: p.read_text() for p in merged.glob("run_manifest*.json")} + assert actual == manifests + + +@pytest.fixture +def download_mock(monkeypatch, tmp_path): + from fastfill import download + from huggingface_hub.hf_api import RepoFile + + sources = {"first": ("owner/one", "first"), "second": ("owner/one", "second"), + "third": ("owner/two", "third")} + api = Mock() + api.list_repo_tree.side_effect = lambda repo, **kwargs: [ + RepoFile(path=f"{folder}/data.json", size=2, oid="0" * 40) + for owner, folder in sources.values() if owner == repo] + monkeypatch.setattr(download, "SOURCES", sources) + monkeypatch.setattr(download, "HfApi", Mock(return_value=api)) + monkeypatch.setattr(sys, "argv", ["download", "--out", str(tmp_path), "--workers", "1"]) + return download, api + + +@pytest.mark.parametrize("failures", [{"first/data.json"}, {"first/data.json", "second/data.json", "third/data.json"}]) +def test_download_failures_exit_nonzero_after_all_sources(download_mock, monkeypatch, failures): + download, api = download_mock + fetch = Mock(side_effect=lambda repo, path, local: path not in failures) + monkeypatch.setattr(download, "fetch", fetch) + with pytest.raises(SystemExit) as error: + download.main() + assert error.value.code == 1 + assert fetch.call_count == 3 + assert api.list_repo_tree.call_count == 2 + + +def test_download_success_and_size_resume(download_mock, monkeypatch, tmp_path): + download, _ = download_mock + cached = tmp_path / "owner__one/first/data.json" + cached.parent.mkdir(parents=True) + cached.write_text("{}") + fetch = Mock(return_value=True) + monkeypatch.setattr(download, "fetch", fetch) + assert download.main() in (None, 0) + assert [c.args[1] for c in fetch.call_args_list] == ["second/data.json", "third/data.json"] + + +def test_download_module_entrypoint_exits_after_retries(monkeypatch, tmp_path): + import huggingface_hub + from huggingface_hub.hf_api import RepoFile + + api = Mock() + api.list_repo_tree.return_value = [RepoFile(path="SAGE10k/data.json", size=2, oid="0" * 40)] + monkeypatch.setattr(huggingface_hub, "HfApi", Mock(return_value=api)) + fetch = Mock(side_effect=OSError("offline fixture failure")) + monkeypatch.setattr(huggingface_hub, "hf_hub_download", fetch) + monkeypatch.setattr("time.sleep", lambda _: None) + monkeypatch.setattr(sys, "argv", ["download", "--out", str(tmp_path), "--only", "SAGE-10k", "--workers", "1"]) + with pytest.raises(SystemExit) as error: + runpy.run_path(str(ROOT / "fastfill/download.py"), run_name="__main__") + assert error.value.code == 1 + assert fetch.call_count == 8 + + +def test_tiny_real_cpu_merge(tmp_path): + import torch + from peft import LoraConfig, PeftModel, get_peft_model + from tokenizers import Tokenizer + from tokenizers.models import WordLevel + from tokenizers.pre_tokenizers import Whitespace + from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedTokenizerFast, Qwen3Config, Qwen3ForCausalLM + + torch.set_num_threads(1) + torch.manual_seed(42) + backend = Tokenizer(WordLevel({"": 0, "": 1, "hello": 2, "world": 3, "": 4}, unk_token="")) + backend.pre_tokenizer = Whitespace() + tok = PreTrainedTokenizerFast(tokenizer_object=backend, unk_token="", eos_token="") + tok.add_special_tokens({"additional_special_tokens": [""]}) + tok.chat_template = "{% for m in messages %}{{ m['content'] }}{% endfor %}" + cfg = Qwen3Config(vocab_size=len(tok), hidden_size=16, intermediate_size=32, num_hidden_layers=1, + num_attention_heads=2, num_key_value_heads=1, head_dim=8, max_position_embeddings=32, + eos_token_id=1, rope_theta=1000000.) + base, adapter, merged = tmp_path / "base", tmp_path / "run/final", tmp_path / "merged" + model = Qwen3ForCausalLM(cfg).eval() + model.save_pretrained(base) + tok.save_pretrained(base) + lora = get_peft_model(model, LoraConfig(task_type="CAUSAL_LM", r=2, lora_alpha=4, target_modules=["q_proj"])) + with torch.no_grad(): + for name, parameter in lora.named_parameters(): + if "lora_" in name: + parameter.fill_(0.125) + lora.save_pretrained(adapter) + tok.pad_token = tok.eos_token + tok.save_pretrained(adapter) + for name in ("run_manifest.json", "run_manifest.resume1.json"): + (adapter.parent / name).write_text('{"fixture":true}\n') + reference = PeftModel.from_pretrained( + AutoModelForCausalLM.from_pretrained(base, dtype=torch.bfloat16, local_files_only=True), + adapter, torch_dtype=torch.bfloat16).eval() + ids = torch.tensor([[2, 3, 1]]) + with torch.no_grad(): + before = reference(input_ids=ids).logits.float() + load_merge().apply_lora(str(base), str(merged), str(adapter)) + reloaded = AutoModelForCausalLM.from_pretrained(merged, dtype=torch.bfloat16, local_files_only=True).eval() + merged_tok = AutoTokenizer.from_pretrained(merged, local_files_only=True) + with torch.no_grad(): + after = reloaded(input_ids=ids).logits.float() + assert (before - after).abs().max().item() < 0.01 + assert merged_tok.pad_token_id == tok.pad_token_id == 1 + assert merged_tok.get_vocab() == tok.get_vocab() + assert merged_tok.chat_template == tok.chat_template + assert merged_tok(["hello", "hello world"], padding=True, return_tensors="pt")["input_ids"].shape == (2, 2) + saved = json.loads((merged / "config.json").read_text()) + assert saved["rope_theta"] == saved["rope_parameters"]["rope_theta"] == 1000000. + tc = json.loads((merged / "tokenizer_config.json").read_text()) + assert "additional_special_tokens" in tc and not isinstance(tc.get("extra_special_tokens"), list) + assert (merged / "run_manifest.resume1.json").read_text() == '{"fixture":true}\n' diff --git a/code/fastfill/tools/__init__.py b/code/fastfill/tools/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/code/fastfill/tools/ablation_check.py b/code/fastfill/tools/ablation_check.py new file mode 100644 index 0000000000000000000000000000000000000000..0ea09a39d75547187637a1fe03f222e1cf91251f --- /dev/null +++ b/code/fastfill/tools/ablation_check.py @@ -0,0 +1,48 @@ +"""python -m fastfill.tools.ablation_check +The constrained build must differ from the base build only by the `constraints` key of some train rows: +same complete rows (including source, flags and all message metadata) after removing only that key from +the constrained user JSON; dev/test files byte-identical (one evaluation serves both).""" +import hashlib +from itertools import zip_longest +import json +import sys + + +def _normalized(row, remove_constraints=False): + """Keep the entire row; normalize only the intended user JSON payload.""" + user = json.loads(row["messages"][1]["content"]) + if remove_constraints: + user = {k: v for k, v in user.items() if k != "constraints"} + normalized = {**row, "messages": [ + {**m, "content": user} if i == 1 else m for i, m in enumerate(row["messages"])]} + # JSON booleans and numbers must not compare equal (Python's True == 1 does). + return json.dumps(normalized, sort_keys=True, ensure_ascii=False, separators=(",", ":")) + + +def _sha256(path): + digest = hashlib.sha256() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + digest.update(chunk) + return digest.digest() + + +def main(base, cons): + diff = n_rows = n_cons = 0 + with open(f"{base}/train.jsonl", encoding="utf-8") as a, open(f"{cons}/train.jsonl", encoding="utf-8") as b: + for lx, ly in zip_longest(a, b): + assert lx is not None and ly is not None, "train row counts differ" + x, y = json.loads(lx), json.loads(ly) + n_rows += 1 + n_cons += "constraints" in json.loads(y["messages"][1]["content"]) + diff += _normalized(x) != _normalized(y, remove_constraints=True) + same_eval = all(_sha256(f"{base}/{f}") == _sha256(f"{cons}/{f}") + for f in ("dev.jsonl", "test.jsonl", "dev_rooms.jsonl", "test_rooms.jsonl", + "dev_constrained_rooms.jsonl", "test_constrained_rooms.jsonl")) + print(json.dumps({"train_rows": n_rows, "rows_differing_beyond_constraints": diff, + "constrained_rows_in_" + cons.rstrip("/").split("/")[-1]: n_cons, "eval_files_identical": same_eval})) + assert diff == 0 and same_eval, "not a clean ablation" + + +if __name__ == "__main__": + main(sys.argv[1], sys.argv[2]) diff --git a/code/fastfill/tools/qa_report.py b/code/fastfill/tools/qa_report.py new file mode 100644 index 0000000000000000000000000000000000000000..391dfd20c5d9dbf2fd68dd18620ad15cbd93d224 --- /dev/null +++ b/code/fastfill/tools/qa_report.py @@ -0,0 +1,420 @@ +"""Streaming QA: python -m fastfill.tools.qa_report -> /QA.json. + +--ir/--data are equivalent named arguments. No dataset version is assumed. +Checks written outputs against MANIFEST, constraint schema/references/truth on the saved +answers, closure, geometry, selection and leakage. Replays actual SFT text using the +manifest's augmentation, constraint and dropout settings. Only compact UID/digest/key +indexes and counters are retained, never whole datasets or whole files in memory. +Replay checks persisted rows, not the build's global split/dedup/cap selection decisions. +Writes the report before exiting 1 on mandatory failures; trainer drift and geometry +rates remain informational rather than requiring every reference to be pristine. +""" +import argparse +import collections +import copy +import hashlib +from itertools import combinations +import json +import math +from pathlib import Path + +from shapely.geometry import Polygon + +from fastfill.build import _code_files, extract_constraints, prep, rng_for +from fastfill.evaluate import inside, score +from fastfill.interface import snap_inside +from fastfill.scene import canonical, messages, parse, rot90 +from fastfill.split import content_key, layout_key +from fastfill.validate import check, holds + + +SPLITS = ("train", "dev", "test") +OUTPUTS = ("train.jsonl", "dev.jsonl", "test.jsonl", "dev_rooms.jsonl", "test_rooms.jsonl", + "dev_constrained_rooms.jsonl", "test_constrained_rooms.jsonl", "stats.json") +BUILD_CODE = {"__init__.py", "build.py", "anchors.py", "scene.py", "split.py", "validate.py"} +DEFAULT_FLAGS = {"oob_objects", "overlapping_furniture", "fixed_collision"} +DEFAULTS = dict(reject_flagged=[], keep_duplicates=False, min_objects=1, max_vertices=0, + sources=None, source_anchors={}, boundary_types=["polygon", "hull"], + anchors=["floor", "object"], oob_tol=.1, hidden_max=.3, + no_rot90=False, constraint_frac=0., field_dropout=.2, desc_dropout=.5) + + +def rows(path): + """Missing outputs are disclosed by provenance; never materialize JSONL files.""" + if Path(path).is_file(): + with open(path, encoding="utf-8") as f: + for number, line in enumerate(f, 1): + try: + yield json.loads(line) + except ValueError as exc: + raise ValueError(f"{path}:{number}: invalid JSON") from exc + + +def _file_info(path): + digest, lines, last = hashlib.sha256(), 0, b"" + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + digest.update(chunk) + lines += chunk.count(b"\n") + last = chunk[-1:] + return {"sha256": digest.hexdigest(), "lines": lines + int(bool(last) and last != b"\n")} + + +def _scoped_code(name): + return not any(p.startswith(("review", ".", "__pycache__")) or p == "tests" + for p in Path(name).parts[:-1]) + + +def _provenance(man, ir, data): + here = Path(__file__).resolve().parents[1] + current = {str(p.relative_to(here)): _file_info(p)["sha256"] for p in _code_files(here)} + recorded = {k: v for k, v in man.get("code_sha256", {}).items() if _scoped_code(k)} + diff = sorted(k for k in current.keys() | recorded.keys() if current.get(k) != recorded.get(k)) + qa = {"build code differing from MANIFEST (must be empty)": [k for k in diff if k in BUILD_CODE], + "other code changed since the build (does not affect the data)": [k for k in diff if k not in BUILD_CODE], + "legacy MANIFEST code entries outside build provenance scope": + sorted(k for k in man.get("code_sha256", {}) if not _scoped_code(k))} + ir_hashes = man.get("ir_sha256") + qa["IR files differing from MANIFEST (must be empty)"] = ( + sorted(k for k, v in ir_hashes.items() + if not (ir / k).is_file() or _file_info(ir / k)["sha256"] != v) + if ir_hashes else "no ir_sha256 in MANIFEST") + declared = man.get("files") or {} + differences = {} + for name in sorted(set(declared) | set(OUTPUTS)): + path = data / name + if not path.is_file(): + differences[name] = {"missing": True} + continue + if name not in declared: + continue # Disclose missing declarations separately, not as verified hashes. + actual = _file_info(path) + mismatch = {k: {"expected": declared[name].get(k), "actual": v} + for k, v in actual.items() if declared[name].get(k) != v} + if mismatch: + differences[name] = mismatch + qa["output files differing from MANIFEST (must be empty)"] = differences + qa["output files missing from MANIFEST (must be empty)"] = sorted(set(OUTPUTS) - set(declared)) + return qa + + +def _layout(row): + user = json.loads(row["messages"][1]["content"]) + objs = [{"id": o["id"], "category": o["id"].rsplit("_", 1)[0], "size": o["size"]} for o in user["objects"]] + pl, err = parse(row["messages"][2]["content"], [o["id"] for o in objs]) + # SFT fixed boxes use integer degrees; every IR geometry consumer uses radians. + fixed = [{**f, "yaw": math.radians(f["yaw"])} for f in user.get("fixed", [])] + room = {"boundary": user["boundary"], "boundary_type": user["boundary_type"], + "height": user.get("height"), "fixed": fixed, "objects": objs} + lay = {**room, "objects": [{**o, "pos": pl[o["id"]]["pos"], "yaw": pl[o["id"]]["yaw"], + "parent": pl[o["id"]]["on"]} for o in objs if o["id"] in pl]} + return user, room, pl, lay, err + + +def _valid(row): + """Bounds/support/ceiling validity of the saved answer, using displayed inputs.""" + _, _, _, lay, err = _layout(row) + return err is None and check(lay)["valid"] + + +def _constraint_error(c, ids): + arity = {"on": 3, "faces": 3, "near": 4, "against_wall": 2, "between": 4} + if not isinstance(c, list) or not c or not isinstance(c[0], str) or len(c) != arity.get(c[0]): + return "schema" + refs = c[1:3] if c[0] == "near" else c[1:] + if not all(isinstance(v, str) and v for v in refs): + return "schema" + if c[0] == "near": + d = c[3] + try: + good = type(d) in (int, float) and math.isfinite(d) and d >= 0 + except OverflowError: + good = False + if not good: + return "schema" + return None if all(v in ids for v in refs) else "reference" + + +def _constraints(constraints, objects, boundary, ids=None, answer_error=None): + counts, failures = collections.Counter(), collections.Counter() + if not isinstance(constraints, list): + return counts, collections.Counter(schema=1), 0, 1 + index = {o["id"]: o for o in objects} + ids = set(index) if ids is None else ids + ok = 0 + for c in constraints: + kind = c[0] if isinstance(c, list) and c and isinstance(c[0], str) else "malformed" + counts[kind] += 1 + error = _constraint_error(c, ids) + if not error and answer_error: + error = "answer" + if not error and not holds(c, objects, boundary, index): + error = "false" + if error: + failures[error] += 1 + else: + ok += 1 + return counts, failures, ok, len(constraints) + + +def _percentiles(hist): + total = sum(hist.values()) + out = dict.fromkeys(("p50", "p90", "p99", "max")) + for key, fraction in zip(out, (.5, .9, .99, 1.)): + rank, n = min(int(fraction * total), total - 1), 0 + for value, count in sorted(hist.items()): + n += count + if n > rank: + out[key] = value + break + return out + + +def _train_metrics(records): + closure, fixed, objects, gone = (collections.Counter() for _ in range(4)) + types, failures = collections.Counter(), collections.Counter() + n = constrained = fixed_answered = raw = repaired = used = valid = cons_ok = cons_total = 0 + for row in records: + n += 1 + user, room, pl, lay, err = _layout(row) + closure["ok" if err is None else err] += 1 + fixed_answered += any(f["id"] in pl for f in user.get("fixed", [])) + fixed[len(user.get("fixed", []))] += 1 + objects[len(user["objects"])] += 1 + constrained += "constraints" in user + c, f, ok, total = _constraints(user.get("constraints", []), lay["objects"], lay["boundary"], + {o["id"] for o in room["objects"]}, err) + types.update(c); failures.update(f) + cons_ok += ok; cons_total += total + # Exactly train.select's default truthy-flag filter and alphabetical reason precedence. + hit = sorted(k for k, v in (row.get("flags") or {}).items() if v and k in DEFAULT_FLAGS) + if hit: + gone["flag:" + hit[0]] += 1 + else: + used += 1 + gone["weight"] += 0 + valid += err is None and check(lay)["valid"] + if err is None: + floor = Polygon(user["boundary"]) + raw += inside(lay, floor) + adjusted = copy.deepcopy(pl) + snap_inside(adjusted, room, floor) + repaired += inside({**lay, "objects": [{**o, "pos": adjusted[o["id"]]["pos"]} + for o in lay["objects"]]}, floor) + return {"train_rows_with_constraints": constrained, "object_set_closure": dict(closure), + "fixed_boxes_answered (must be 0)": fixed_answered, + "train reference inside walls within 1 mm": {"raw": f"{raw}/{n}", "repaired": f"{repaired}/{n}"}, + "train rows used by default": {"rows": used, "left_out": dict(gone), "reference_valid": f"{valid}/{used}"}, + "train fixed boxes": {"rows_with_fixed": n - fixed[0], "per_row": _percentiles(fixed)}, + "train objects per row": _percentiles(objects), "constraints_by_type": {"train": dict(sorted(types.items()))}, + "constraint_reference_validation": {"train": f"{cons_ok}/{cons_total}"}, + "constraint_failures": {"train": dict(failures)}} + + +def _eval_metrics(data, qa): + qa["reference legality raw vs repaired"] = {} + metrics = {"complete_rate": None, "valid_rate": "valid", "valid_rate_repaired": "valid_repaired", + "inside_1mm_rate": "inside_1mm", "inside_1mm_rate_repaired": "inside_1mm_repaired", + "rooms_with_fixed_collision": "coll_fixed", "rooms_with_severe": "coll_severe"} + for split in ("dev", "test"): + types, failures = collections.Counter(), collections.Counter() + ok = total = 0 + for room in rows(data / f"{split}_constrained_rooms.jsonl"): + c, f, k, n = _constraints(room.get("constraints", []), room["objects"], room["boundary"]) + types.update(c); failures.update(f); ok += k; total += n + qa["constraints_by_type"][split] = dict(sorted(types.items())) + qa["constraint_reference_validation"][split] = f"{ok}/{total}" + qa["constraint_failures"][split] = dict(failures) + groups = collections.defaultdict(collections.Counter) + for room in rows(data / f"{split}_rooms.jsonl"): + r = score(room, None) + counts = groups["held_out" if r["held_out"] else "in_dist"] + counts["rooms"] += 1 + for name, key in metrics.items(): + counts[name] += r["error"] is None if key is None else bool(r.get(key, False)) + for group, counts in groups.items(): + qa["reference legality raw vs repaired"][f"{split}/{group}"] = { + "rooms": counts["rooms"], **{k: counts[k] / counts["rooms"] for k in metrics}} + + +def _digest(value): + # Outer JSON formatting is irrelevant; message content strings are compared exactly. + return hashlib.sha256(json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode()).digest() + + +def _fingerprints(row): + msgs = row["messages"] + return tuple(_digest(v) for v in ( + *[msgs[i] if i < len(msgs) else None for i in range(3)], msgs[3:], + {k: v for k, v in row.items() if k != "messages"})) + + +def _replay(room, split, args): + cons, with_desc = None, True + if split == "train": + aug, s2, drop = (rng_for(room["uid"], k) for k in ("aug", "s2", "dropout")) + c = canonical(room if args.no_rot90 else rot90(room, aug.randrange(4))) + cons = extract_constraints(c, s2) if s2.random() < args.constraint_frac else None + if drop.random() < args.field_dropout: + c = {**c, "room_type": None} + if drop.random() < args.field_dropout: + c = {**c, "height": None} + with_desc = drop.random() >= args.desc_dropout + else: + c = canonical(room) + return {"uid": room["uid"], "source": room["source"], "flags": room["meta"]["flags"], + "messages": messages(c, cons, with_desc=with_desc)} + + +def _record_mismatch(report, split, uid, reasons): + report["failed"] += 1 + report["mismatches"].update(reasons) + if len(report["examples"]) < 15: + report["examples"].append({"split": split, "uid": uid, "reasons": reasons}) + + +def _ir_checks(ir, args, saved): + keys = {s: collections.defaultdict(set) for s in SPLITS} + held, found = set(), set() + src_rows, ir_seen = collections.Counter(), collections.Counter() + report = {"matched": 0, "failed": 0, "mismatches": collections.Counter(), "examples": [], + "scope": "Exact messages and row metadata, using current code and MANIFEST settings; " + "code/IR/output provenance discrepancies are reported separately."} + for path in sorted(ir.glob("*.jsonl")): + if args.sources is not None and path.stem not in args.sources: + continue + for raw in rows(path): + uid = raw["uid"] + if uid not in saved: + continue + ir_seen[uid] += 1 + if uid in found: + continue # First match is checked; ambiguous source UIDs are reported separately. + found.add(uid) + room, why = prep(raw, args) + expected = {} + for split, digests in saved[uid]: + if room is None: + _record_mismatch(report, split, uid, ["prep"]) + continue + if split not in expected: + expected[split] = _fingerprints(_replay(room, split, args)) + names = ("system", "user", "assistant", "extra_messages", "metadata") + reasons = [k for k, actual, target in zip(names, digests, expected[split]) if actual != target] + if reasons: + _record_mismatch(report, split, uid, reasons) + else: + report["matched"] += 1 + src_rows[(split, path.stem)] += 1 + if room is None: + continue + if room.get("meta", {}).get("eval_only"): + held.add(path.stem) + fingerprints = (("furniture keys", content_key(room)), ("layout keys", layout_key(room))) + for split in {s for s, _ in saved[uid]}: + keys[split]["worlds"].add(room["group"]) + keys[split]["aliases"].update(room.get("meta", {}).get("group_aliases") or []) + for name, key in fingerprints: + if key: + keys[split][name].add(key) + for uid in saved.keys() - found: + for split, _ in saved[uid]: + _record_mismatch(report, split, uid, ["missing_ir"]) + leak = {f"{a}&{b} {name}": len(keys[a][name] & keys[b][name]) + for a, b in combinations(SPLITS, 2) for name in ("worlds", "aliases", "furniture keys", "layout keys")} + leak.update({"held-out sources in train (rows)": sum(n for (s, src), n in src_rows.items() if s == "train" and src in held), + "rows not reproducible from IR": report["failed"], + "rows whose uid is missing from the IR": sum(len(saved[u]) for u in saved.keys() - found), + "uids written more than once (within or across splits)": sum(len(v) > 1 for v in saved.values()), + "persisted uids found more than once in IR": sum(n > 1 for n in ir_seen.values())}) + return {"worlds": {s: len(keys[s]["worlds"]) for s in SPLITS}, + "leakage (all must be 0)": leak, "SFT reproducibility": report} + + +def _arguments(): + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("ir", nargs="?") + ap.add_argument("data", nargs="?") + ap.add_argument("--ir", dest="ir_option") + ap.add_argument("--data", dest="data_option") + a = ap.parse_args() + for key in ("ir", "data"): + positional, option = getattr(a, key), getattr(a, key + "_option") + if positional and option and positional != option: + ap.error(f"conflicting {key} paths") + value = option or positional + if not value: + ap.error(f"{key} path is required") + setattr(a, key, value) + return a + + +def _rejections(data): + if not (data / "stats.json").is_file(): + return None + with open(data / "stats.json", encoding="utf-8") as f: + stats = json.load(f) + return { + src: {k.replace("rejected:", ""): v for k, v in d.items() + if k.startswith(("rejected:", "flag:", "fixed:")) or k in + ("scanned", "dev", "test", "fixed_items", "rooms_with_fixed", "objects_max")} + | {"kept_train": d.get("train_S1", 0) + d.get("train_S2", 0)} for src, d in stats.items()} + + +def _mandatory_failures(qa): + """Gate explicit invariants, not descriptive geometry rates or non-build code drift.""" + failures = [key for key in ( + "build code differing from MANIFEST (must be empty)", + "IR files differing from MANIFEST (must be empty)", + "output files differing from MANIFEST (must be empty)", + "output files missing from MANIFEST (must be empty)", + "fixed_boxes_answered (must be 0)", + ) if qa[key]] + if any(n for error, n in qa["object_set_closure"].items() if error != "ok"): + failures.append("object_set_closure") + if any(any(counts.values()) for counts in qa["constraint_failures"].values()): + failures.append("constraint_failures") + if any(qa["leakage (all must be 0)"].values()): + failures.append("leakage (all must be 0)") + if qa["SFT reproducibility"]["failed"]: + failures.append("SFT reproducibility") + return failures + + +def main(): + a = _arguments() + ir, data = Path(a.ir), Path(a.data) + with open(data / "MANIFEST.json", encoding="utf-8") as f: + man = json.load(f) + settings = {**DEFAULTS, **man["args"]} + anchors = settings["source_anchors"] + settings["source_anchors"] = dict(anchors) if isinstance(anchors, dict) else { + k: v.split(",") for k, v in (x.split("=", 1) for x in anchors)} + args = argparse.Namespace(**settings) + qa = {"build": a.data, **_provenance(man, ir, data), + "MANIFEST defaults used": {k: v for k, v in DEFAULTS.items() if k not in man["args"]}} + saved, counts = collections.defaultdict(list), collections.Counter() + + def indexed(split): + for row in rows(data / f"{split}.jsonl"): + counts[split] += 1 + saved[row["uid"]].append((split, _fingerprints(row))) + yield row + + qa.update(_train_metrics(indexed("train"))) + for split in ("dev", "test"): + for _ in indexed(split): + pass + qa["rows"] = {s: counts[s] for s in SPLITS} + _eval_metrics(data, qa) + qa.update(_ir_checks(ir, args, saved)) + qa["rejections_by_source"] = _rejections(data) + qa["mandatory_failures"] = _mandatory_failures(qa) + with open(data / "QA.json", "w", encoding="utf-8") as f: + json.dump(qa, f, indent=1) + print(json.dumps({k: v for k, v in qa.items() if k != "rejections_by_source"}, indent=1)) + return int(bool(qa["mandatory_failures"])) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/code/fastfill/tools/visualize.py b/code/fastfill/tools/visualize.py new file mode 100644 index 0000000000000000000000000000000000000000..a7794f229c7ac5f045e770ceebbd827a81b0c03a --- /dev/null +++ b/code/fastfill/tools/visualize.py @@ -0,0 +1,146 @@ +"""FastFill 2D Floor Plan & Layout Visualizer. + +Renders room boundaries, fixed elements (doors/windows/stairs), and object placements +with bounding boxes, IDs, labels, and front orientation arrows. + +Usage: + # 1. Visualize a sample from test.jsonl: + python -m fastfill.tools.visualize --data data/v3.1/test.jsonl --index 0 --out room_0.png + + # 2. Render the built-in synthetic demonstration: + python -m fastfill.tools.visualize --out demo.png + +Requires matplotlib. --data accepts SFT message rows; evaluator predictions are not accepted by this CLI. +""" +import argparse +import json +import math +import os +import matplotlib +matplotlib.use("Agg") +import matplotlib.pyplot as plt +from matplotlib.patches import Polygon as MPoly +import numpy as np + + +def draw_layout(room, placements, save_path="layout.png"): + fig, ax = plt.subplots(figsize=(9, 9), facecolor="#f8f9fa") + ax.set_facecolor("#ffffff") + + # 1. Draw boundary + b = room.get("boundary", []) + if b: + bx, by = zip(*(b + [b[0]])) + ax.plot(bx, by, color="#1a1a1a", lw=3.5, label="Boundary / Wall", zorder=2) + ax.fill(bx, by, facecolor="#f1f3f5", alpha=0.5, zorder=1) + + # 2. Draw fixed elements (doors, windows, columns, stairs) + for f in room.get("fixed", []): + x, y = f["pos"][:2] + sx, sy = f["size"][:2] + yaw = math.radians(f.get("yaw", 0)) + corners = np.array([ + [-sx/2, -sy/2], + [ sx/2, -sy/2], + [ sx/2, sy/2], + [-sx/2, sy/2] + ]) + R = np.array([[math.cos(yaw), -math.sin(yaw)], [math.sin(yaw), math.cos(yaw)]]) + rot = corners @ R.T + np.array([x, y]) + ax.add_patch(MPoly(rot, facecolor="#dee2e6", edgecolor="#495057", lw=1.5, hatch="//", zorder=3)) + ax.text(x, y, f.get("id", "fixed"), fontsize=7, ha="center", va="center", color="#212529", zorder=4) + + # 3. Draw object placements + pmap = {p["id"]: p for p in placements} + colors = ["#4ea8de", "#56cfe1", "#72efdd", "#80ffdb", "#48bfe3", "#64dfdf"] + + for idx, obj in enumerate(room.get("objects", [])): + oid = obj["id"] + if oid not in pmap: + continue + p = pmap[oid] + x, y = p["pos"][:2] + sx, sy = obj["size"][:2] + yaw = math.radians(p.get("yaw", 0)) + + corners = np.array([ + [-sx/2, -sy/2], + [ sx/2, -sy/2], + [ sx/2, sy/2], + [-sx/2, sy/2] + ]) + R = np.array([[math.cos(yaw), -math.sin(yaw)], [math.sin(yaw), math.cos(yaw)]]) + rot = corners @ R.T + np.array([x, y]) + + c = colors[idx % len(colors)] + ax.add_patch(MPoly(rot, facecolor=c, edgecolor="#1d3557", lw=2, alpha=0.85, zorder=5)) + + # Front arrow pointing along local +X + arrow_len = min(sx, sy) * 0.45 + dx = math.cos(yaw) * arrow_len + dy = math.sin(yaw) * arrow_len + ax.arrow(x, y, dx, dy, head_width=0.08, head_length=0.08, fc="#e63946", ec="#e63946", zorder=6) + + # Label + label = f"{oid}\n{p.get('yaw', 0)}°" + ax.text(x, y, label, fontsize=8, ha="center", va="center", fontweight="bold", color="#1d3557", zorder=7) + + room_type = room.get("room_type", "Room Layout") + ax.set_title(f"FastFill Scene Layout: {room_type} ({len(placements)} objects)", fontsize=13, fontweight="bold", pad=12) + ax.autoscale() + ax.set_aspect("equal") + ax.grid(True, linestyle=":", alpha=0.5, zorder=0) + plt.tight_layout() + + os.makedirs(os.path.dirname(os.path.abspath(save_path)), exist_ok=True) + plt.savefig(save_path, dpi=200) + plt.close() + print(f"[Visualizer] Layout saved to {save_path}") + + +def main(): + parser = argparse.ArgumentParser(description="FastFill Layout Visualizer") + parser.add_argument("--data", default=None, help="Path to jsonl file (train/dev/test)") + parser.add_argument("--index", type=int, default=0, help="Line index to visualize") + parser.add_argument("--out", default="layout.png", help="Output PNG path") + args = parser.parse_args() + + if not args.data: + # Create a synthetic demo room if no data passed + demo_room = { + "room_type": "bedroom", + "boundary": [[0, 0], [4.5, 0], [4.5, 3.8], [0, 3.8]], + "fixed": [ + {"id": "door_1", "size": [0.9, 0.15, 2.0], "pos": [0.8, 0, 0], "yaw": 0}, + {"id": "window_1", "size": [1.5, 0.2, 1.2], "pos": [4.5, 2.0, 1.0], "yaw": 90} + ], + "objects": [ + {"id": "double_bed_1", "size": [2.0, 1.8, 0.8]}, + {"id": "nightstand_1", "size": [0.5, 0.5, 0.5]}, + {"id": "wardrobe_1", "size": [0.6, 1.6, 2.0]}, + {"id": "desk_1", "size": [0.7, 1.2, 0.75]}, + {"id": "chair_1", "size": [0.5, 0.5, 0.8]} + ] + } + demo_placements = [ + {"id": "double_bed_1", "pos": [2.0, 2.7, 0], "yaw": 270}, + {"id": "nightstand_1", "pos": [3.3, 2.7, 0], "yaw": 270}, + {"id": "wardrobe_1", "pos": [0.4, 2.8, 0], "yaw": 0}, + {"id": "desk_1", "pos": [3.8, 0.5, 0], "yaw": 180}, + {"id": "chair_1", "pos": [3.2, 0.5, 0], "yaw": 0} + ] + draw_layout(demo_room, demo_placements, args.out) + return + + with open(args.data, "r", encoding="utf-8") as f: + for i, line in enumerate(f): + if i == args.index: + rec = json.loads(line) + user_msg = json.loads(rec["messages"][1]["content"]) + asst_msg = json.loads(rec["messages"][2]["content"]) + draw_layout(user_msg, asst_msg.get("placements", []), args.out) + break + + +if __name__ == "__main__": + main() diff --git a/code/fastfill/train.py b/code/fastfill/train.py new file mode 100644 index 0000000000000000000000000000000000000000..012fd4abc6d2cef5aea8de2271841396038a99f0 --- /dev/null +++ b/code/fastfill/train.py @@ -0,0 +1,283 @@ +"""LoRA SFT for FastFill v1 (adapted from scripts/sft_train.py). + +Differences from the OptiScene script: pre-split files instead of a random split, loss on the +assistant answer only, over-length samples dropped (never truncated mid-JSON), EOS = <|im_end|> +supervised, Qwen3 thinking disabled in the chat template, final adapter saved; run_manifest.json (the build's +MANIFEST + these arguments) is written into --out so every checkpoint says which data it saw. +Hyper-parameters: LoRA r16/alpha32 as in the OptiScene paper, but lr 1e-4 x 2 epochs (~6k steps at batch 32) instead +of the paper's 5e-6 x 10 epochs, which is a full-fine-tuning rate, ~20x too low for a rank-16 adapter. + +Effective batch = bs x grad_accum x n_gpus; keep it at 32 when changing the GPU count: + 1 GPU: CUDA_VISIBLE_DEVICES=0 python -m fastfill.train --model Qwen/Qwen3-8B --data DATA --out outputs/ff-v1 --bs 4 --grad_accum 8 + 8 GPUs: torchrun --nproc_per_node 8 -m fastfill.train --model Qwen/Qwen3-8B --data DATA --out outputs/ff-v1 --bs 4 --grad_accum 1 +Resume after an interruption: add --resume (continues from the last checkpoint in --out). +Sequence length: --max_len is the model's limit (Qwen3-8B: 40960), not a memory setting; a batch is padded to its +longest sample, so memory follows the data. Measure it before choosing --bs / --grad_accum: + --dry_run token length and object count percentiles with the real tokenizer, the samples + over --max_len listed (they are dropped from training, never truncated) + --mem_test 8192,16384,32768,max one forward/backward/optimizer step per length on one GPU with a batch of --bs rows + padded like a training batch (row 0 at that length, the others at half, see + mem_batch); peak memory printed next to the static part (weights + LoRA params/grads + + AdamW states, read after a warm-up step); 'max' = the longest train or dev row, + because Trainer evaluates dev every --save_steps + --eval_max_len N leave dev rows longer than N tokens out of that evaluation (eval loss then covers + the remaining subset); pick N from the mem_test table +""" +import argparse +import collections +import hashlib +import json +import math +import os + +import torch +from peft import LoraConfig, get_peft_model +from transformers import (AutoModelForCausalLM, AutoTokenizer, DataCollatorForSeq2Seq, Trainer, + TrainingArguments, set_seed) + +from fastfill.scene import prompt_text + + +def end_token(tok): + return "<|im_end|>" if "<|im_end|>" in tok.get_vocab() else tok.eos_token + + +def encode(tok, rows, max_len, chunk=2000): + """Tokenize (prompt, answer) pairs in batches (per-row tokenizer calls are ~300x slower); labels mask the + prompt; rows longer than max_len are dropped, never truncated (a cut JSON answer teaches broken output). + -> (samples, dropped row indices).""" + out, dropped, end = [], [], end_token(tok) + total = len(rows) + for i in range(0, total, chunk): + if total > 5000 and (i // chunk) % 10 == 0: + print(f" [Tokenize] {i}/{total} samples ({i*100//total}%)...", flush=True) + part = rows[i:i + chunk] + ps = tok([prompt_text(tok, m[:-1]) for m in part], add_special_tokens=False)["input_ids"] + ans = tok([m[-1]["content"] + end for m in part], add_special_tokens=False)["input_ids"] + for k, (p, a) in enumerate(zip(ps, ans)): + if len(p) + len(a) > max_len: + dropped.append(i + k) + else: + out.append({"input_ids": p + a, "attention_mask": [1] * (len(p) + len(a)), "labels": [-100] * len(p) + a}) + return out, dropped + + +def select(recs, exclude_flags=(), source_weight=None): + """The sampling layer: rows with any flag in exclude_flags are left out; a row of a source with weight w is + used int(w) times plus once more with probability frac(w) (stable per uid), so w=0.5 keeps half a source and + w=2 doubles it. Default: every row once (the data's own proportions).""" + out, gone = [], collections.Counter() + for r in recs: + hit = {k for k, v in (r.get("flags") or {}).items() if v} & set(exclude_flags) + if hit: + gone["flag:" + sorted(hit)[0]] += 1 + continue + w = (source_weight or {}).get(r.get("source"), 1.0) + u = int(hashlib.sha1(r["uid"].encode()).hexdigest(), 16) % 10 ** 6 / 10 ** 6 + n = int(w) + (u < w - int(w)) + gone["weight"] += n == 0 + out += [r] * n + return out, gone + + +def load(tok, path, max_len, exclude_flags=(), source_weight=None): + print(f"Loading and tokenizing {path}...", flush=True) + recs, gone = select([json.loads(line) for line in open(path)], exclude_flags, source_weight) + rows, dropped = encode(tok, [r["messages"] for r in recs], max_len) + comp = collections.Counter(r.get("source") for r in recs) + print(f"{path}: {len(rows)} samples, dropped {len(dropped)} longer than {max_len} tokens" + + (f", left out {dict(gone)}" if +gone else "") + f"; by source {dict(comp.most_common())}", flush=True) + return rows + + +def length_report(tok, path, max_len): + """Real-tokenizer profile of one file: tokens per sample (total / answer) and objects per room, the samples over + max_len listed by uid so they can be looked at instead of silently dropped.""" + recs = [json.loads(line) for line in open(path)] + rows, over = encode(tok, [r["messages"] for r in recs], float("inf")) + lens = [len(r["input_ids"]) for r in rows] + ans = [sum(l != -100 for l in r["labels"]) for r in rows] + nobj = [len(json.loads(r["messages"][1]["content"])["objects"]) for r in recs] + nfix = [len(json.loads(r["messages"][1]["content"]).get("fixed", [])) for r in recs] + pct = lambda v: {k: sorted(v)[min(int(f * len(v)), len(v) - 1)] for k, f in + (("p50", .5), ("p90", .9), ("p95", .95), ("p99", .99), ("max", 1.0))} + rep = {"file": path, "samples": len(rows), "tokens_total": sum(lens), "tokens": pct(lens), "answer_tokens": pct(ans), + "objects": pct(nobj), "fixed": pct(nfix), + "over_max_len": [{"uid": recs[i]["uid"], "source": recs[i]["source"], "tokens": lens[i], "objects": nobj[i]} + for i in range(len(rows)) if lens[i] > max_len], + "buckets": {f"<= {b}": sum(l <= b for l in lens) for b in (4096, 8192, 16384, 32768, max_len)}} + return rep + + +def mem_batch(tok, longest, spec, bs): + """The batch DataCollatorForSeq2Seq(padding=True) builds for a --bs step whose longest row has the requested length: + 'max' = the longest train/dev row itself, a number = that many tokens (the longest row's tokens repeated, every + token supervised: an upper bound). The other bs-1 rows are the same tokens cut to half and right-padded (pad id, + attention_mask 0, labels -100), so SDPA materializes its (bs,1,L,L) mask as in almost every real batch. + -> {input_ids, attention_mask, labels}: bs lists of L ints each.""" + if spec == "max": + ids, lab = longest["input_ids"], longest["labels"] + else: + ids = (longest["input_ids"] * (int(spec) // len(longest["input_ids"]) + 1))[:int(spec)] + lab = ids + L, h = len(ids), len(ids) // 2 + rows = [(ids, lab)] + [(ids[:h], lab[:h])] * (bs - 1) + return {"input_ids": [i + [tok.pad_token_id] * (L - len(i)) for i, _ in rows], + "attention_mask": [[1] * len(i) + [0] * (L - len(i)) for i, _ in rows], + "labels": [l + [-100] * (L - len(l)) for _, l in rows]} + + +def mem_test(model, tok, train, dev, specs, bs): + """One training step per requested length on the current GPU in the training configuration: a batch of bs rows + padded like the collator's (mem_batch), gradient checkpointing on. 'max' = the longest train or dev row (Trainer + evaluates dev every save_steps). A warm-up step (512 tokens) first allocates the LoRA grads and AdamW states, so + static = weights + LoRA params/grads + AdamW states and the per-length column is what the step adds + (activations, logits, attention mask). OOM at one length moves on to the next.""" + gpu = torch.device("cuda") + model.to(gpu) + model.train() + model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False}) + opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4) + longest = max(train + dev, key=lambda r: len(r["input_ids"])) + print(f"longest train row {max(len(r['input_ids']) for r in train)} tokens, " + f"longest dev row {max(len(r['input_ids']) for r in dev)} tokens (eval set)", flush=True) + + def step(spec): # -> peak bytes of one forward/backward/optimizer step, None on OOM + b = None + torch.cuda.reset_peak_memory_stats() + try: # the batch's own allocation can OOM too + b = {k: torch.tensor(v, device=gpu) for k, v in mem_batch(tok, longest, spec, bs).items()} + model(**b).loss.backward() + opt.step() + return torch.cuda.max_memory_allocated() + except torch.cuda.OutOfMemoryError: + return None + finally: + opt.zero_grad(set_to_none=False) # grads stay resident, as in training + del b + torch.cuda.empty_cache() + + if step("512") is None: # warm-up: creates the LoRA grads and the AdamW moments + print(f"OOM already at 512 tokens x bs {bs} on {torch.cuda.get_device_name()}: the model does not fit this card " + f"in this configuration; no table", flush=True) + return + static = torch.cuda.memory_allocated() + print(f"static (weights + LoRA params/grads + AdamW states) {static / 2 ** 30:.1f} GB", flush=True) + for spec in specs: + n = len(longest["input_ids"]) if spec == "max" else int(spec) + peak = step(spec) + if peak is None: + print(f"len {n:6d} bs {bs} OOM on {torch.cuda.get_device_name()} " + f"({torch.cuda.get_device_properties(gpu).total_memory / 2 ** 30:.0f} GB)", flush=True) + else: + print(f"len {n:6d} bs {bs} peak {peak / 2 ** 30:5.1f} GB = static {static / 2 ** 30:.1f} + " + f"activations/logits/mask {(peak - static) / 2 ** 30:.1f} GB", flush=True) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--model", required=True, help="HF id or local path, e.g. Qwen/Qwen3-8B") + ap.add_argument("--data", required=True, help="dir with train.jsonl / dev.jsonl from fastfill.build") + ap.add_argument("--out", required=True) + ap.add_argument("--max_len", type=int, default=40960, help="model sequence limit; longer samples are dropped") + ap.add_argument("--lr", type=float, default=1e-4) + ap.add_argument("--epochs", type=float, default=2) + ap.add_argument("--bs", type=int, default=4, help="per-device batch size") + ap.add_argument("--grad_accum", type=int, default=8, help="per GPU; divide by the GPU count to keep batch 32") + ap.add_argument("--lora_r", type=int, default=16) + ap.add_argument("--lora_alpha", type=int, default=32) + ap.add_argument("--full_ft", action="store_true", help="full fine-tuning instead of LoRA (use with --deepspeed)") + ap.add_argument("--deepspeed", default=None) + ap.add_argument("--attn", default="sdpa", help="sdpa | flash_attention_2") + ap.add_argument("--save_steps", type=int, default=500) + ap.add_argument("--report_to", default="none") + ap.add_argument("--dry_run", action="store_true", help="tokenize and print the length profile only (no GPU)") + ap.add_argument("--mem_test", default=None, help="e.g. 8192,16384,32768,max: measure peak memory per length, then exit") + ap.add_argument("--exclude_flags", nargs="*", default=["oob_objects", "overlapping_furniture", "fixed_collision"], + help="train rows carrying any of these quality flags are left out (see the row's 'flags'). Default: " + "the three whose reference answer serve.py would refuse (objects > 10 cm through the walls; " + "copies of one piece through each other; furniture through a door / column / stair box). " + "Pass --exclude_flags with no value to train on all") + ap.add_argument("--source_weight", nargs="*", default=[], + help="SOURCE=w: use each train row of SOURCE w times on average (0.5 halves it, 2 doubles it)") + ap.add_argument("--eval_max_len", type=int, default=0, + help="dev rows longer than this are left out of the eval at every save_steps (0 = keep all)") + ap.add_argument("--resume", action="store_true", help="continue from the last checkpoint in --out") + a = ap.parse_args() + local_rank = int(os.environ.get("LOCAL_RANK", 0)) + if torch.cuda.is_available(): + torch.cuda.set_device(local_rank) + a.source_weight = {k: float(v) for k, v in (x.split("=") for x in a.source_weight)} + known = {json.loads(line)["source"] for line in open(f"{a.data}/train.jsonl")} if a.source_weight else set() + if set(a.source_weight) - known: + raise SystemExit(f"--source_weight: unknown source(s) {sorted(set(a.source_weight) - known)}; known: {sorted(known)}") + + print(f"Loading tokenizer from {a.model}...", flush=True) + tok = AutoTokenizer.from_pretrained(a.model) + if tok.pad_token is None: + tok.pad_token = tok.eos_token + if a.dry_run: + for split in ("train", "dev", "test"): + rep = length_report(tok, f"{a.data}/{split}.jsonl", a.max_len) + print(json.dumps(rep, indent=1)) + train = load(tok, f"{a.data}/train.jsonl", a.max_len, a.exclude_flags, a.source_weight) + print(tok.decode(train[0]["input_ids"])) + print("supervised part:", repr(tok.decode([t for t in train[0]["labels"] if t != -100]))) + return + train = load(tok, f"{a.data}/train.jsonl", a.max_len, a.exclude_flags, a.source_weight) + dev = load(tok, f"{a.data}/dev.jsonl", a.max_len) + if a.eval_max_len: # protect the eval at every save_steps; eval loss then covers a subset + kept = [r for r in dev if len(r["input_ids"]) <= a.eval_max_len] + print(f"eval set: {len(dev) - len(kept)} dev rows longer than {a.eval_max_len} tokens left out, {len(kept)} kept", flush=True) + dev = kept + if int(os.environ.get("RANK", 0)) == 0 and not a.mem_test: # provenance of this run (build MANIFEST + args) + os.makedirs(a.out, exist_ok=True) + man = f"{a.data}/MANIFEST.json" + rec = {"args": vars(a), "data_manifest": json.load(open(man)) if os.path.exists(man) else None} + path = f"{a.out}/run_manifest.json" + if a.resume and os.path.exists(path): # a resume is recorded beside the run it continues; a new run replaces + k = 1 + while os.path.exists(f"{a.out}/run_manifest.resume{k}.json"): + k += 1 + path = f"{a.out}/run_manifest.resume{k}.json" + json.dump(rec, open(path, "w"), indent=1) + + model = AutoModelForCausalLM.from_pretrained( + a.model, dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32, attn_implementation=a.attn) + model.config.use_cache = False + set_seed(42) # before get_peft_model: same LoRA init every run (v1.0 vs v1.1 pairing) + if not a.full_ft: + model.enable_input_require_grads() # needed for gradient checkpointing with frozen embeddings + model = get_peft_model(model, LoraConfig( + task_type="CAUSAL_LM", r=a.lora_r, lora_alpha=a.lora_alpha, lora_dropout=0.05, + target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"])) + model.print_trainable_parameters() + if a.mem_test: + mem_test(model, tok, train, dev, a.mem_test.split(","), a.bs) + return + + # 3% warmup as an integer step count: transformers 4.x rejects fractional warmup_steps, >=5.15 has no warmup_ratio + # one process that sees several GPUs runs DataParallel: its batch is bs x n_gpu, like DDP's + world = int(os.environ.get("WORLD_SIZE", 0)) or max(1, torch.cuda.device_count()) + total_steps = math.ceil(math.ceil(len(train) / (a.bs * world)) / a.grad_accum) * a.epochs + # batch similar lengths together (~1/3 less padding); the argument was renamed in transformers 5.14 + by_length = ({"group_by_length": True} if "group_by_length" in TrainingArguments.__dataclass_fields__ + else {"train_sampling_strategy": "group_by_length"}) + args = TrainingArguments( + output_dir=a.out, per_device_train_batch_size=a.bs, per_device_eval_batch_size=1, # dev pads to 40k tokens + gradient_accumulation_steps=a.grad_accum, learning_rate=a.lr, num_train_epochs=a.epochs, **by_length, + lr_scheduler_type="cosine", warmup_steps=max(1, int(0.03 * total_steps)), weight_decay=0.0, + bf16=torch.cuda.is_available(), + gradient_checkpointing=True, gradient_checkpointing_kwargs={"use_reentrant": False}, + logging_steps=10, eval_strategy="steps", eval_steps=a.save_steps, save_strategy="steps", + save_steps=a.save_steps, save_total_limit=3, remove_unused_columns=False, + report_to=a.report_to, deepspeed=a.deepspeed, ddp_find_unused_parameters=False) + trainer = Trainer(model=model, args=args, train_dataset=train, eval_dataset=dev, + data_collator=DataCollatorForSeq2Seq(tok, padding=True, label_pad_token_id=-100)) + trainer.train(resume_from_checkpoint=True if a.resume else None) + trainer.save_model(f"{a.out}/final") + tok.save_pretrained(f"{a.out}/final") + + +if __name__ == "__main__": + main() diff --git a/code/fastfill/v2/.coveragerc b/code/fastfill/v2/.coveragerc new file mode 100644 index 0000000000000000000000000000000000000000..1dccc0430d6dab3930bd661d05f59ab30c2bfec8 --- /dev/null +++ b/code/fastfill/v2/.coveragerc @@ -0,0 +1,6 @@ +[run] +source = fastfill.v2 +omit = */fastfill/v2/tests/* + +[report] +fail_under = 80 diff --git a/code/fastfill/v2/README.md b/code/fastfill/v2/README.md new file mode 100644 index 0000000000000000000000000000000000000000..f190b8cfc5d228348d6ebf8e03483812172f39ff --- /dev/null +++ b/code/fastfill/v2/README.md @@ -0,0 +1,854 @@ +# FastFill v2 — experimental joint geometry generator + +Main task: **room geometry + object requests + fixed objects + available support +and relation constraints** predicts one target local full size, bottom-center and +yaw per stable requested ID. Continuous heads provide structural supervision; +Hungarian matching is restricted to legal exchangeable groups. Deployment design +then resolves actual assets, reconciles actual geometry and validates the whole +scene before atomic Host commit. Current code implements the offline reference +contracts; real mesh/physics/Solver checkers and a persistent WorldEdge Host +adapter remain integration work. +The separate three-field `reference_extent` view is a simplified-input ablation +that exports bbox JSON and GLB/SVG without an asset library. It does not replace +the main full-condition task or its deployment acceptance requirements. +Qwen encodes condition tokens only. An external bidirectional object decoder +cross-attends to all condition hidden states and produces continuous geometry. +Text SFT is an independent token-CE comparison. Neither requires MinkowskiEngine. + +The architecture follows the 2026-10-05 design. The direct minimal-input boundary +is an independent ablation. It is **not a validated method**. +No production Qwen training, quality improvement, real Host deployment, mesh or +physical simulation success is claimed. Existing `fastfill/train.py`, +`evaluate.py`, `serve.py`, and v3/v3.1/v3.2 dataset/experiment lineage are retained. +Model version v2 is separate from the older dataset's v3-series numbering. + +The final mesh consumer is **RoomGenBench**. `predict --export-dir` now saves +canonical `condition.json` / `layout.json` alongside SceneSpec, asset registry, +GLB/SVG and diagnostics. `python -m fastfill.v2.roomgenbench` accepts any such +handoff and an independent output directory, optionally consuming generated +GLB/JSON sidecars. It preserves declared support, constraints and fixed bbox +proxies, counts failed/fallback/missing assets, and records native/fitted sizes. +See the [downstream contract and commands](../../docs/fastfill-v2-roomgenbench-interface-20261006.md). +This mesh-fit adapter does not run learned generation or certify physics/Host. + +The formal runs (round 2) are `configs/qwen3_8b_main_4gpu_regression.json` +(4 ranks x B1 x K24, regression position head) and +`configs/qwen3_8b_main_3gpu_grid.json` (3 ranks x B1 x K32, grid_residual head): +both global batch 96, 3,887 updates (3 epochs of 124,375 scenes), warmup 117, +validation and checkpoint every 500. They inherit everything else from +`configs/qwen3_8b_main_world7.json` (the earlier seven-rank candidate). Pass +them explicitly with the qualified dataset; generic defaults remain +development configurations. Launch commands are in the round-2 section below. Latest audit, source pins and publication state are +in the [design audit](../../docs/fastfill-v2-design-audit-20261006.md). + +## What is implemented + +| Module | Behavior | +|---|---| +| `schema.py`, `geometry.py`, `batch.py` | Strict finite-number/ID protocol, condition-only normalization/token spans, masks, fixed geometry, complete-sample budget rejection | +| `model.py` | Qwen-family condition backbone (LoRA/full/frozen), request-bound slots, bidirectional self-attention + cross-attention, positive exponential size, position (`model.position_head` regression or grid_residual), yaw logits/residuals, checkpoints | +| `matching.py`, `losses.py` | Fixed identity or explicitly certified within-group Hungarian (groups from `validity.exchangeable_group`), detached assignment, differentiable original tensors, complete-field masks, L1 / smooth-L1 selectable position and size terms, symmetry-aware paired yaw loss, box-symmetric size/yaw candidates for `size_axis_swap_allowed`, per-term sums and counts for window logging | +| `boxes.py`, `regularizers.py` | Optional BEV oriented convex-hull GIoU, normalized collision-volume and convex-room boundary losses; no implicit 3D GIoU | +| `audit.py`, `adapters.py`, `data.py`, `legacy_build.py`, `legacy_bridge.py`, `legacy_evidence.py` | Source inventory, immutable migration of the selected v3.2 corpus with inherited splits and evidence-based field masks; explicit raw MultiScan audit/smoke adapter | +| `qualified_data.py` | New full-condition main derivative with pinned parent/source hashes, D1/D2 qualification changes, fixed-identity fallback for groups losing complete geometry, per-member change records and unchanged target numbers | +| `train.py`, `text_sft.py`, `evaluate.py` | Joint optimizer with warmup/cosine schedule, decoder LR, seeded augmentation, resume and per-checkpoint export; assistant-only text CE; checkpoint inference, all-request failures, legal matching reference metrics, mean-predictor baselines, collapse metrics and separated asset/system metrics | +| `direct_layout.py`, `bbox_visualization.py`, `minimal_data.py`, `multisource_data.py` | Independent three-field ablation, explicit reference-extent semantics, source-audited partial-mask projection, canonical/RoomGenBench JSON with polygon-wall shell, door/window openings, declared/inferred/unknown `place` and benchmark-convention shared `asset_key`, no-asset bbox GLB/SVG and proxy diagnostics | +| `predict.py` | Main `--condition` checkpoint inference; separate `--request` ablation and optional offline catalog execution; `--max-length` defaults to the checkpoint's training value | +| `runtime.py`, `validation.py`, `serve.py` | Catalog resolver contract, target/actual separation, pivot-to-canonical transforms, verified support propagation, bounded reselect/translation repair, atomic in-memory Host reference | + +The latest [objective update](../../docs/fastfill-v2-review3-objective.md) adds enabled-loss preflight and a globally counted accumulation-window optimizer guard. + +## 2026-10-06 audit fixes + +Contract changes from the three-group audit (schema, serialization, yaw policy, matching, losses, trainer, evaluation, RoomGenBench handoff): + +- **Row schema.** `condition.objects[i]` carries only `id`, `category`, `description` and optional `support_parent`. Exchangeable groups moved to `validity.exchangeable_group` (`list[str|null]`, target order) and `validity.yaw_symmetry_order` (`list[int]`, 1 = semantic front, 2 = axis mod pi) is always present. `io.read_samples` migrates old rows (`schema.migrate_legacy_row`) once; `validate_condition` rejects `exchangeable_group` in objects. +- **Condition text.** `batch.condition_segments` renders `schema_version, room, constraints, objects` in that fixed order (objects last, so causal object tokens see the room), drops `room.boundary_quality` and every `_`-prefixed key; object span tagging and the prompt header are unchanged. The direct request path therefore renders the same room fields as a training rectangle. +- **Yaw policy at build.** Default `--front-policy axis`: yaw valid = upright, finite, `front_known`; `yaw_symmetry_order` 1 for MultiScan, 2 elsewhere; round 2: `validity.size_axis_swap_allowed` (box may equally be (sx, sy, yaw) or (sy, sx, yaw + pi/2)) is true for InternScenes arkit/3rscan/mp3d/scannet, InteriorGS, HSSD200 categories containing `chair` and MultiScan categories containing `bed`, and those objects carry order 4 (old rows migrate to all false). Floor support comes from the source annotation only: an `anchor = floor` (not inferred) object on a known floor is declared `support_parent: floor` and its target z snapped when within 2 cm (`field_evidence.legacy_z_snap_applied_to_target`), otherwise left undeclared with free z; the manifest counts `floor_declarations_written` / `floor_declaration_z_snapped` / `floor_declaration_skipped_floating` and `source_size_axis_swap_allowed_objects`. `strict` and `legacy-convention` remain selectable (`legacy_bridge.FRONT_POLICIES`). Descriptions are the source `desc` or the category (`provenance.descriptions = source_desc_or_category`). Groups need complete **position** only; signature = category/description/support_parent. +- **Qualification.** Any size axis < 3 mm masks the whole size vector (`degenerate_axis_lt_3mm`); a target above `room.height_m + 0.05` keeps labels and the declared height and is flagged in `provenance.height_conflict = {objects, max_excess_m}` (journal field `provenance.height_conflict`, reason `target_exceeds_declared_height_flag_only`; round 2 reverts the round-1 height drop); the five RoomGenBench SAGE layouts move to the test split with `provenance.holdout_reason = roomgenbench_benchmark_room`. `legacy_verify` / `multisource_verify` recompute `source_yaw_valid_objects`, `source_yaw_symmetry_order_counts`, `exchangeable_group_counts`, `yaw_policy`, `descriptions` and accept the holdout. +- **Matching and losses.** `match_batch` reads `batch["exchangeable_group"]` (per request slot). Group cost = `alpha_position * L1(position)` always, plus `alpha_size * L1(log size)` only when every member has a complete size label; position-incomplete groups keep fixed identity. `loss.position_type` / `loss.size_type` in {`l1`, `smooth_l1`} (default `l1`), `loss.smooth_l1_beta` (default 1). Main-config weights 1.0 / 0.6 / 0.08 / 7.0 (round 2 caps yaw_reg at 2.0) come from the [calibration note](../../docs/fastfill-v2-loss-calibration-20261006.md). The criterion exposes `term_sums` / `term_counts` so window logs are count-weighted means over every microbatch. +- **Trainer.** New sections `optimizer` (`warmup_steps` default 3 % of steps, `schedule` cosine to 10 % LR, `decoder_lr`), `augmentation` (`rotate90`, `mirror`, `shuffle_objects`, `drop_constraints_p`, `drop_support_p`, `category_only_description_p`, round 2 `minimal_form_p`; training rows only, seeded per (seed, epoch, row)), `validation.exclude_flags` (default the three training flags), `training.resume`, `training.export_model_every_checkpoint`. Unknown keys raise. `--resume ` continues into a new output directory with identical config/data; each checkpoint writes `state-step-/` and `model-step-/`; `train.log` tees stdout; bf16 windows with nonfinite gradients are skipped and counted; a room-centre / category-median-size / uniform-yaw baseline is logged before training. +- **Evaluation.** `report.json` adds `baselines` (room centre, category mean position, category median size, uniform yaw; leave-one-out or `--baseline-fit `), `collapse` (predicted vs ground-truth overlap, stacking, wall distance, central fraction), `by_source` and `yaw_error_rad_by_symmetry_order`. +- **RoomGenBench export.** `room.walls` from the floor polygon (0.1 m thick, height = room height or 2.7), `doors` / `windows` from fixed objects attached to the nearest wall, `place` / `support_status` (round 2, `direct_layout.infer_support`): a declared support parent or hard `on` gives `floor` / `on_object` / `wall` with `declared`; otherwise `floor` when the bottom is within 2 cm of a known floor height (none when `floor_known` is false, e.g. `reference_extent`), else `on_object` for the highest strictly-lower predicted box whose top is within 3 cm and whose footprint contains the object's centre and whose declared or earlier inferred support chain does not lead back to the object (no `A -> B -> A` from a declaration), both `inferred`; else (round 10) `wall` when the bottom is over 0.15 m above a known floor, a footprint side lies within `validation.WALL_GAP_M` (0.1 m) of a known boundary and no box lies under the object's centre, `inferred`; anything else `unknown`. `assets.jsonl` and assembly receipts carry the same `place` and `support_status`; `asset_key = slug(type)[:24] + "_" + sha1(description)[:8]` shared by identical type+description (benchmark convention), per-instance dimensions in the scene. Tests need the reference assembler at `RoomGenBench/` or `FASTFILL_ROOMGENBENCH_ROOT`. See the [server setup guide](../../docs/fastfill-v2-server-start.md) for the supplied H20Z host. The historical [review2 audit](../../docs/fastfill-v2-review-20261006.md) separates repaired defects, data provenance and remaining training limits. + +Review repairs on top of the above (same day): + +- **Augmentation.** `rotate90` / `mirror` snap rigid-motion float noise (`8.8 - 1.1` renders as `7.7`, full-precision source values are kept), and `train._Rows` falls back to the preflighted row when an augmented condition exceeds `max_length` (`run_manifest.augmentation_fallbacks`). After `drop_constraints_p` / `drop_support_p` / `category_only_description_p` fire, exchangeable groups are recomputed with the build rule (`matching.group_labels`): requests made textually identical by a drop are matched by geometry instead of fixed identity; build-time groups whose wider candidate fails certification keep their members under a `kept_