diff --git a/.gitignore b/.gitignore index e1a699ac37f449b6c2b99d720554c9122433b2de..2b9e3f89fc9eb37a000f327b65544dd6a9ed87c6 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,6 @@ *.onnx + +weights/ +vfms/ +__pycache__/ +.gradio/ diff --git a/LICENSE_CorridorKey2.0 b/LICENSE_CorridorKey2.0 new file mode 100644 index 0000000000000000000000000000000000000000..c88c7ebf9d29f59f64da5ccded10e1f0a6b269f6 --- /dev/null +++ b/LICENSE_CorridorKey2.0 @@ -0,0 +1,597 @@ +CORRIDOR KEY LICENCE +======================================================================= + +Version 2.0 + +Copyright (c) Corridor Digital LLC. All rights reserved. + + +ADDITIONAL TERMS AND CONDITIONS +======================================================================= + +This work is licensed under the Creative Commons +Attribution-NonCommercial-ShareAlike 4.0 International Public License +(CC BY-NC-SA 4.0), the full text of which is included below, subject +to the following additional terms and conditions. These additional +terms supplement the CC BY-NC-SA 4.0 licence and take precedence +where they conflict. + +"This tool" means the CorridorKey software in this repository and +the CorridorKey model weights distributed for use with it. + +By exercising any rights to the Licensed Material, You accept and +agree to be bound by both these Additional Terms and the +CC BY-NC-SA 4.0 Public License. + + +1. PERMITTED USE + + a. You may use this tool for any lawful purpose, including for + processing images as part of a commercial project, provided + that such use complies with the restrictions set out below + and the terms of the CC BY-NC-SA 4.0 licence. + + b. The images and video You produce with this tool are Yours. + The Licensor claims no rights in them, and You may use them + commercially. + + +2. RESTRICTIONS + + In addition to the restrictions set out in the CC BY-NC-SA 4.0 + licence, the following restrictions apply: + + a. NO REPACKAGING OR RESALE + You may not repackage, redistribute, sublicense, or sell + this tool, in whole or in part, as a standalone product or + as part of a competing product, and You may not charge for + access to this tool's code or model weights. + + b. NO PAID API OR INFERENCE SERVICES + You may not use this tool to provide inference as a paid + API service, whether directly or indirectly. This includes, + but is not limited to, offering access to this tool behind + a paywall, subscription model, or usage-based billing + system. + + c. DERIVATIVES MUST BE FREELY ACCESSIBLE + You may build on this tool, but any project, fork, + derivative work, or modified version that You release and + that incorporates this tool must be made available to the + public free of charge. Voluntary donations and + pay-what-you-want pricing are permitted, provided that + paying nothing is always an available option. + + d. COMMERCIAL SOFTWARE INTEGRATION + If you operate a commercial software package or inference + service and wish to incorporate this tool into your + software, you must obtain a separate written agreement + from the Licensor. Please contact: + + contact@corridordigital.com + + +3. LARGE ORGANISATIONS + + a. If You are, or are using this tool on behalf of, a Legal + Entity whose total gross revenue exceeded US $20,000,000 + (twenty million US dollars) in its most recently completed + fiscal year, the permissions in Section 1 do not apply + unless that Legal Entity has purchased a seat of the + CorridorKey plugin from the Licensor for each individual + who uses this tool. + + b. "Legal Entity" means the entity using this tool together + with all other entities that control, are controlled by, + or are under common control with it. Revenue is measured + across the whole Legal Entity. + + c. Each seat of the CorridorKey plugin includes a licence + for one individual to download and use this tool and its + model weights. Seats are available at: + + https://corridorkey.com + + +4. ATTRIBUTION + + a. In addition to the attribution requirements set out in + Section 3(a) of the CC BY-NC-SA 4.0 licence, You must + include the name "CorridorKey" in any attribution notice. + + b. Any fork, derivative work, variation, improvement, or + subsequent release of this tool must retain the + "CorridorKey" name in a reasonably prominent position. + + c. You may interchange "Corridor Key" for "CorridorKey" as + needed. + + +5. SHARE-ALIKE + + Any variations, improvements, derivative works, or modified + versions of this tool that are publicly released must be + distributed under this same licence, including these Additional + Terms, or a licence with substantially equivalent terms. + + +6. THIRD-PARTY COMPONENTS + + This tool depends on third-party code and model weights that + are not owned by the Licensor and are not covered by this + licence. Each remains under its own licence, as listed in + THIRD_PARTY_NOTICES.md. + + +7. DISCLAIMER + + This licence is provided on an "as-is" basis. The Licensor makes + no representations or warranties regarding the enforceability or + legal sufficiency of these terms. You are encouraged to seek + independent legal advice if you have questions about the scope + or applicability of this licence. + + +8. CONTRIBUTIONS + + By submitting any contribution (including source code, + documentation, or images) to this repository, You agree that: + + a. Your contribution is provided under the terms of this + Corridor Key Licence. + + b. You grant the Licensor (Corridor Digital LLC) a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, + irrevocable license to use, reproduce, prepare derivative + works of, publicly display, and sublicense your + contribution, including for commercial purposes. + + c. You represent that you are the legal owner of the + contribution or have the authority to submit it under + these terms. + +======================================================================= + + +https://creativecommons.org/licenses/by-nc-sa/4.0/ +https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode.txt + +======================================================================= + +Attribution-NonCommercial-ShareAlike 4.0 International + +======================================================================= + +Creative Commons Corporation ("Creative Commons") is not a law firm and +does not provide legal services or legal advice. Distribution of +Creative Commons public licenses does not create a lawyer-client or +other relationship. Creative Commons makes its licenses and related +information available on an "as-is" basis. Creative Commons gives no +warranties regarding its licenses, any material licensed under their +terms and conditions, or any related information. Creative Commons +disclaims all liability for damages resulting from their use to the +fullest extent possible. + +Using Creative Commons Public Licenses + +Creative Commons public licenses provide a standard set of terms and +conditions that creators and other rights holders may use to share +original works of authorship and other material subject to copyright +and certain other rights specified in the public license below. The +following considerations are for informational purposes only, are not +exhaustive, and do not form part of our licenses. + + Considerations for licensors: Our public licenses are + intended for use by those authorized to give the public + permission to use material in ways otherwise restricted by + copyright and certain other rights. Our licenses are + irrevocable. Licensors should read and understand the terms + and conditions of the license they choose before applying it. + Licensors should also secure all rights necessary before + applying our licenses so that the public can reuse the + material as expected. Licensors should clearly mark any + material not subject to the license. This includes other CC- + licensed material, or material used under an exception or + limitation to copyright. More considerations for licensors: + wiki.creativecommons.org/Considerations_for_licensors + + Considerations for the public: By using one of our public + licenses, a licensor grants the public permission to use the + licensed material under specified terms and conditions. If + the licensor's permission is not necessary for any reason--for + example, because of any applicable exception or limitation to + copyright--then that use is not regulated by the license. Our + licenses grant only permissions under copyright and certain + other rights that a licensor has authority to grant. Use of + the licensed material may still be restricted for other + reasons, including because others have copyright or other + rights in the material. A licensor may make special requests, + such as asking that all changes be marked or described. + Although not required by our licenses, you are encouraged to + respect those requests where reasonable. More considerations + for the public: + wiki.creativecommons.org/Considerations_for_licensees + +======================================================================= + +Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International +Public License + +By exercising the Licensed Rights (defined below), You accept and agree +to be bound by the terms and conditions of this Creative Commons +Attribution-NonCommercial-ShareAlike 4.0 International Public License +("Public License"). To the extent this Public License may be +interpreted as a contract, You are granted the Licensed Rights in +consideration of Your acceptance of these terms and conditions, and the +Licensor grants You such rights in consideration of benefits the +Licensor receives from making the Licensed Material available under +these terms and conditions. + + +Section 1 -- Definitions. + + a. Adapted Material means material subject to Copyright and Similar + Rights that is derived from or based upon the Licensed Material + and in which the Licensed Material is translated, altered, + arranged, transformed, or otherwise modified in a manner requiring + permission under the Copyright and Similar Rights held by the + Licensor. For purposes of this Public License, where the Licensed + Material is a musical work, performance, or sound recording, + Adapted Material is always produced where the Licensed Material is + synched in timed relation with a moving image. + + b. Adapter's License means the license You apply to Your Copyright + and Similar Rights in Your contributions to Adapted Material in + accordance with the terms and conditions of this Public License. + + c. BY-NC-SA Compatible License means a license listed at + creativecommons.org/compatiblelicenses, approved by Creative + Commons as essentially the equivalent of this Public License. + + d. Copyright and Similar Rights means copyright and/or similar rights + closely related to copyright including, without limitation, + performance, broadcast, sound recording, and Sui Generis Database + Rights, without regard to how the rights are labeled or + categorized. For purposes of this Public License, the rights + specified in Section 2(b)(1)-(2) are not Copyright and Similar + Rights. + + e. Effective Technological Measures means those measures that, in the + absence of proper authority, may not be circumvented under laws + fulfilling obligations under Article 11 of the WIPO Copyright + Treaty adopted on December 20, 1996, and/or similar international + agreements. + + f. Exceptions and Limitations means fair use, fair dealing, and/or + any other exception or limitation to Copyright and Similar Rights + that applies to Your use of the Licensed Material. + + g. License Elements means the license attributes listed in the name + of a Creative Commons Public License. The License Elements of this + Public License are Attribution, NonCommercial, and ShareAlike. + + h. Licensed Material means the artistic or literary work, database, + or other material to which the Licensor applied this Public + License. + + i. Licensed Rights means the rights granted to You subject to the + terms and conditions of this Public License, which are limited to + all Copyright and Similar Rights that apply to Your use of the + Licensed Material and that the Licensor has authority to license. + + j. Licensor means the individual(s) or entity(ies) granting rights + under this Public License. + + k. NonCommercial means not primarily intended for or directed towards + commercial advantage or monetary compensation. For purposes of + this Public License, the exchange of the Licensed Material for + other material subject to Copyright and Similar Rights by digital + file-sharing or similar means is NonCommercial provided there is + no payment of monetary compensation in connection with the + exchange. + + l. Share means to provide material to the public by any means or + process that requires permission under the Licensed Rights, such + as reproduction, public display, public performance, distribution, + dissemination, communication, or importation, and to make material + available to the public including in ways that members of the + public may access the material from a place and at a time + individually chosen by them. + + m. Sui Generis Database Rights means rights other than copyright + resulting from Directive 96/9/EC of the European Parliament and of + the Council of 11 March 1996 on the legal protection of databases, + as amended and/or succeeded, as well as other essentially + equivalent rights anywhere in the world. + + n. You means the individual or entity exercising the Licensed Rights + under this Public License. Your has a corresponding meaning. + + +Section 2 -- Scope. + + a. License grant. + + 1. Subject to the terms and conditions of this Public License, + the Licensor hereby grants You a worldwide, royalty-free, + non-sublicensable, non-exclusive, irrevocable license to + exercise the Licensed Rights in the Licensed Material to: + + a. reproduce and Share the Licensed Material, in whole or + in part, for NonCommercial purposes only; and + + b. produce, reproduce, and Share Adapted Material for + NonCommercial purposes only. + + 2. Exceptions and Limitations. For the avoidance of doubt, where + Exceptions and Limitations apply to Your use, this Public + License does not apply, and You do not need to comply with + its terms and conditions. + + 3. Term. The term of this Public License is specified in Section + 6(a). + + 4. Media and formats; technical modifications allowed. The + Licensor authorizes You to exercise the Licensed Rights in + all media and formats whether now known or hereafter created, + and to make technical modifications necessary to do so. The + Licensor waives and/or agrees not to assert any right or + authority to forbid You from making technical modifications + necessary to exercise the Licensed Rights, including + technical modifications necessary to circumvent Effective + Technological Measures. For purposes of this Public License, + simply making modifications authorized by this Section 2(a) + (4) never produces Adapted Material. + + 5. Downstream recipients. + + a. Offer from the Licensor -- Licensed Material. Every + recipient of the Licensed Material automatically + receives an offer from the Licensor to exercise the + Licensed Rights under the terms and conditions of this + Public License. + + b. Additional offer from the Licensor -- Adapted Material. + Every recipient of Adapted Material from You + automatically receives an offer from the Licensor to + exercise the Licensed Rights in the Adapted Material + under the conditions of the Adapter's License You apply. + + c. No downstream restrictions. You may not offer or impose + any additional or different terms or conditions on, or + apply any Effective Technological Measures to, the + Licensed Material if doing so restricts exercise of the + Licensed Rights by any recipient of the Licensed + Material. + + 6. No endorsement. Nothing in this Public License constitutes or + may be construed as permission to assert or imply that You + are, or that Your use of the Licensed Material is, connected + with, or sponsored, endorsed, or granted official status by, + the Licensor or others designated to receive attribution as + provided in Section 3(a)(1)(A)(i). + + b. Other rights. + + 1. Moral rights, such as the right of integrity, are not + licensed under this Public License, nor are publicity, + privacy, and/or other similar personality rights; however, to + the extent possible, the Licensor waives and/or agrees not to + assert any such rights held by the Licensor to the limited + extent necessary to allow You to exercise the Licensed + Rights, but not otherwise. + + 2. Patent and trademark rights are not licensed under this + Public License. + + 3. To the extent possible, the Licensor waives any right to + collect royalties from You for the exercise of the Licensed + Rights, whether directly or through a collecting society + under any voluntary or waivable statutory or compulsory + licensing scheme. In all other cases the Licensor expressly + reserves any right to collect such royalties, including when + the Licensed Material is used other than for NonCommercial + purposes. + + +Section 3 -- License Conditions. + +Your exercise of the Licensed Rights is expressly made subject to the +following conditions. + + a. Attribution. + + 1. If You Share the Licensed Material (including in modified + form), You must: + + a. retain the following if it is supplied by the Licensor + with the Licensed Material: + + i. identification of the creator(s) of the Licensed + Material and any others designated to receive + attribution, in any reasonable manner requested by + the Licensor (including by pseudonym if + designated); + + ii. a copyright notice; + + iii. a notice that refers to this Public License; + + iv. a notice that refers to the disclaimer of + warranties; + + v. a URI or hyperlink to the Licensed Material to the + extent reasonably practicable; + + b. indicate if You modified the Licensed Material and + retain an indication of any previous modifications; and + + c. indicate the Licensed Material is licensed under this + Public License, and include the text of, or the URI or + hyperlink to, this Public License. + + 2. You may satisfy the conditions in Section 3(a)(1) in any + reasonable manner based on the medium, means, and context in + which You Share the Licensed Material. For example, it may be + reasonable to satisfy the conditions by providing a URI or + hyperlink to a resource that includes the required + information. + 3. If requested by the Licensor, You must remove any of the + information required by Section 3(a)(1)(A) to the extent + reasonably practicable. + + b. ShareAlike. + + In addition to the conditions in Section 3(a), if You Share + Adapted Material You produce, the following conditions also apply. + + 1. The Adapter's License You apply must be a Creative Commons + license with the same License Elements, this version or + later, or a BY-NC-SA Compatible License. + + 2. You must include the text of, or the URI or hyperlink to, the + Adapter's License You apply. You may satisfy this condition + in any reasonable manner based on the medium, means, and + context in which You Share Adapted Material. + + 3. You may not offer or impose any additional or different terms + or conditions on, or apply any Effective Technological + Measures to, Adapted Material that restrict exercise of the + rights granted under the Adapter's License You apply. + + +Section 4 -- Sui Generis Database Rights. + +Where the Licensed Rights include Sui Generis Database Rights that +apply to Your use of the Licensed Material: + + a. for the avoidance of doubt, Section 2(a)(1) grants You the right + to extract, reuse, reproduce, and Share all or a substantial + portion of the contents of the database for NonCommercial purposes + only; + + b. if You include all or a substantial portion of the database + contents in a database in which You have Sui Generis Database + Rights, then the database in which You have Sui Generis Database + Rights (but not its individual contents) is Adapted Material, + including for purposes of Section 3(b); and + + c. You must comply with the conditions in Section 3(a) if You Share + all or a substantial portion of the contents of the database. + +For the avoidance of doubt, this Section 4 supplements and does not +replace Your obligations under this Public License where the Licensed +Rights include other Copyright and Similar Rights. + + +Section 5 -- Disclaimer of Warranties and Limitation of Liability. + + a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE + EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS + AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF + ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, + IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, + WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR + PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, + ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT + KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT + ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. + + b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE + TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, + NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, + INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, + COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR + USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN + ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR + DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR + IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. + + c. The disclaimer of warranties and limitation of liability provided + above shall be interpreted in a manner that, to the extent + possible, most closely approximates an absolute disclaimer and + waiver of all liability. + + +Section 6 -- Term and Termination. + + a. This Public License applies for the term of the Copyright and + Similar Rights licensed here. However, if You fail to comply with + this Public License, then Your rights under this Public License + terminate automatically. + + b. Where Your right to use the Licensed Material has terminated under + Section 6(a), it reinstates: + + 1. automatically as of the date the violation is cured, provided + it is cured within 30 days of Your discovery of the + violation; or + + 2. upon express reinstatement by the Licensor. + + For the avoidance of doubt, this Section 6(b) does not affect any + right the Licensor may have to seek remedies for Your violations + of this Public License. + + c. For the avoidance of doubt, the Licensor may also offer the + Licensed Material under separate terms or conditions or stop + distributing the Licensed Material at any time; however, doing so + will not terminate this Public License. + + d. Sections 1, 5, 6, 7, and 8 survive termination of this Public + License. + + +Section 7 -- Other Terms and Conditions. + + a. The Licensor shall not be bound by any additional or different + terms or conditions communicated by You unless expressly agreed. + + b. Any arrangements, understandings, or agreements regarding the + Licensed Material not stated herein are separate from and + independent of the terms and conditions of this Public License. + + +Section 8 -- Interpretation. + + a. For the avoidance of doubt, this Public License does not, and + shall not be interpreted to, reduce, limit, restrict, or impose + conditions on any use of the Licensed Material that could lawfully + be made without permission under this Public License. + + b. To the extent possible, if any provision of this Public License is + deemed unenforceable, it shall be automatically reformed to the + minimum extent necessary to make it enforceable. If the provision + cannot be reformed, it shall be severed from this Public License + without affecting the enforceability of the remaining terms and + conditions. + + c. No term or condition of this Public License will be waived and no + failure to comply consented to unless expressly agreed to by the + Licensor. + + d. Nothing in this Public License constitutes or may be interpreted + as a limitation upon, or waiver of, any privileges and immunities + that apply to the Licensor or You, including from the legal + processes of any jurisdiction or authority. + +======================================================================= + +Creative Commons is not a party to its public +licenses. Notwithstanding, Creative Commons may elect to apply one of +its public licenses to material it publishes and in those instances +will be considered the “Licensor.” The text of the Creative Commons +public licenses is dedicated to the public domain under the CC0 Public +Domain Dedication. Except for the limited purpose of indicating that +material is shared under a Creative Commons public license or as +otherwise permitted by the Creative Commons policies published at +creativecommons.org/policies, Creative Commons does not authorize the +use of the trademark "Creative Commons" or any other trademark or logo +of Creative Commons without its prior written consent including, +without limitation, in connection with any unauthorized modifications +to any of its public licenses or any other arrangements, +understandings, or agreements concerning use of licensed material. For +the avoidance of doubt, this paragraph does not form part of the +public licenses. + +Creative Commons may be contacted at creativecommons.org. diff --git a/README.md b/README.md index 42e324d60144311ad1d95cebe248a84317c6d8c5..074d822c5d5775f94fc7c2c93a9c6e90b2d3f33a 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,10 @@ sdk_version: 6.9.0 app_file: app.py python_version: "3.10" pinned: false +startup_duration_timeout: 1h +license: other +license_name: corridor-key-licence-2.0 +license_link: LICENSE_CorridorKey2.0 tags: - green-screen - background-removal @@ -27,32 +31,86 @@ short_description: Remove green/blue screen from video, even glass & hair Remove green or blue screen backgrounds from video. Handles transparent objects (glass, water, cloth) that traditional chroma key cannot. -Based on [CorridorKey](https://github.com/nikopueringer/CorridorKey) by Corridor Digital. +Pick the model version in the UI. **v2 is the default.** -## Inference Paths +| | v2 (default) | v1 | +|---|---|---| +| Model | [CorridorKey 2.0](https://github.com/nikopueringer/CorridorKey2.0) | [CorridorKey v1.0](https://github.com/nikopueringer/CorridorKey) | +| Runs on | GPU only (ZeroGPU) | GPU, or CPU through ONNX | +| Frames | Temporal: sees a window of frames together | One frame at a time | +| Mask | None needed, the model finds the subject | BiRefNet / classical pre-mask | +| Input size | Clip is downscaled on CPU to 540p, 720p or 1080p first | Model runs at a fixed 1024 or 2048 | -- **GPU (ZeroGPU H200)**: PyTorch GreenFormer with batched inference (batch 32 at 1024, batch 16 at 2048) +## v2 + +CorridorKey 2.0 is a two-stage transformer. Stage 1 sees frames together (full spatio-temporal attention at 1/64 resolution, Swin-3D at 1/16) using frozen features from three foundation models: +C-RADIO (semantics), MoGe-2 (geometry) and RVM (motion). Stage 2 turns that context plus the raw RGB into a full-resolution alpha and foreground, one frame at a time. + +How this Space runs it: + +1. The clip is downscaled on CPU with ffmpeg to the chosen quality. +2. One ZeroGPU call runs the unmodified v2 runtime. +3. Previews are encoded on CPU. The zip holds the alpha, foreground and composite videos at the working resolution. + +Measured end to end on an RTX 5090 (GPU phase, warm, 89 frames): 540p 19s, 720p 28s, 1080p 48-54s, and 720p with 267 frames 68s. The first call after the model is loaded takes longer. +The GPU time requested from ZeroGPU is estimated from those numbers, so a clip that does not fit gets an error naming the limit instead of being cut off. +The status line reports the GPU that actually ran. + +**Temporal context**: Auto picks a window of 80 frames up to 720p and 24 frames at 1080p. An 80-frame window costs about the same as 8 frames up to 720p, but at 1080p it is about 2.5x slower, which is why Auto narrows it there. Peak VRAM depends on resolution and window size, not clip length (about 6 GB at 720p with 8 frames, 13 GB with 80, 25 GB at 1080p with 80). + +Native 4K is not offered: the example clip is 4K and takes about 160s on an RTX 5090, over the 120s limit. + +## v1 inference paths + +- **GPU (ZeroGPU)**: PyTorch GreenFormer with batched inference (batch 32 at 1024, batch 16 at 2048) - **CPU (fallback)**: ONNX Runtime sequential inference (batch 1) -## Pipeline +### v1 pipeline 1. **BiRefNet** - Generates coarse foreground mask (ONNX, or fast classical HSV for green/blue screens) 2. **CorridorKey GreenFormer** - Refines alpha matte + extracts clean foreground (PyTorch on GPU, ONNX on CPU) -3. **GPU Postprocessing** - Despill, despeckle (connected components), resize — all on GPU via torchvision, single CPU transfer at the end +3. **GPU Postprocessing** - Despill, despeckle (connected components), resize, all on GPU via torchvision, single CPU transfer at the end -## GPU Optimizations +### v1 GPU optimizations -- **Full GPU pipeline**: preprocessing (resize + normalize) and postprocessing (despill, clean_matte, resize) stay on device — avoids CPU↔GPU round-trips per batch +- **Full GPU pipeline**: preprocessing (resize + normalize) and postprocessing (despill, clean_matte, resize) stay on device, avoiding CPU↔GPU round-trips per batch - **TF32 tensor cores**: `torch.set_float32_matmul_precision('high')` for FP32 postprocessing ops -- **AOTI compilation with torch.inductor + triton cudagraphs** (native CUDA kernels, fused ops, replays entire kernel sequence without CPU-GPU sync overhead) don't benefit GreenFormer: tested max-autotune (118s, 0 triton kernels) and reduce-overhead (36s compile + 48s graph recording = 84s for 5% speedup). Small feature maps (112-896ch) are cublas-optimal, not triton-friendly. Disabled on ZeroGPU — eager at 0.32s/frame beats 84s+ overhead. torch.compile still available for local GPU +- **AOTI compilation with torch.inductor + triton cudagraphs** do not benefit GreenFormer: tested max-autotune (118s, 0 triton kernels) and reduce-overhead (36s compile + 48s graph recording = 84s for 5% speedup). Small feature maps (112-896ch) are cublas-optimal, not triton-friendly. Disabled on ZeroGPU: eager at 0.32s/frame beats 84s+ overhead. torch.compile still available for a local GPU + +**v1 pipeline timing** (89 frames, batch 32 @ 1024px model res): CPU mask 22s → GPU load 5s → inference 29s → write 15s → stitch 9s ≈ 80s total, 49s GPU. The model always processes at 1024x1024 or 2048x2048 regardless of input resolution + +## CLI + +v2 is the default. v1 needs `--version v1`, and v1-only options are rejected without it. -**Pipeline timing** (89 frames, batch 32 @ 1024px model res): CPU mask 22s → GPU load 5s → inference 29s → write 15s → stitch 9s ≈ 80s total, 49s GPU. Model always processes at 1024x1024 or 2048x2048 regardless of input resolution +```bash +python app.py --input clip.mp4 # v2 +python app.py --input clip.mp4 --quality 1080p --temporal 24 # v2 options +python app.py --input clip.mp4 --version v1 --resolution 2048 # v1 +python app.py --input clip.mp4 --version v1 --device cpu # v1 on CPU +``` + +v2 options: `--quality {540p,720p,1080p}`, `--temporal {auto,1,8,24}`, `--upscale`. +v1 options: `--device`, `--resolution`, `--mask-mode`, `--no-despeckle`, `--despeckle-size`, `--screen-color`. +Shared: `--despill`, `--output`. + +A `--version v1` run does not download the v2 weights. ## API +The v1 endpoint is unchanged. v2 is a separate endpoint. + ### REST API -**Step 1: Submit request** +**Step 1: Submit request (v2)** +```bash +curl -X POST "https://luminia-corridorkey.hf.space/gradio_api/call/process_video_v2" \ + -H "Content-Type: application/json" \ + -d '{"data": [{"path": "https://example.com/clip.mp4", "meta": {"_type": "gradio.FileData"}}, "720p", "auto", 5, false]}' +``` +Arguments: video, quality (`540p`/`720p`/`1080p`), temporal (`auto`/`1`/`8`/`24`), despill 0-10, upscale previews. + +**v1 (unchanged)** ```bash curl -X POST "https://luminia-corridorkey.hf.space/gradio_api/call/process_video" \ -H "Content-Type: application/json" \ @@ -61,7 +119,7 @@ curl -X POST "https://luminia-corridorkey.hf.space/gradio_api/call/process_video **Step 2: Get result** ```bash -curl "https://luminia-corridorkey.hf.space/gradio_api/call/process_video/{event_id}" +curl "https://luminia-corridorkey.hf.space/gradio_api/call/process_video_v2/{event_id}" ``` ### MCP (Model Context Protocol) @@ -77,8 +135,14 @@ curl "https://luminia-corridorkey.hf.space/gradio_api/call/process_video/{event_ } ``` +## Licence + +CorridorKey 2.0 is released under the Corridor Key Licence (a variation of CC BY-NC-SA 4.0 with additional terms), copied in [LICENSE_CorridorKey2.0](LICENSE_CorridorKey2.0). This Space is free to use, keeps the CorridorKey name, and is not a paid service. Organizations above US $20 million annual revenue need a seat of the official plugin ([corridorkey.com](https://corridorkey.com)). Third-party components are listed in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md): C-RADIOv4 (NVIDIA Open Model License), MoGe-2 (MIT), RVM (CC-BY 4.0). + ## Credits -- [CorridorKey](https://github.com/nikopueringer/CorridorKey) by Niko Pueringer / Corridor Digital (synced to `8a4e4b4`, 2026-05-01) +- [CorridorKey 2.0](https://github.com/nikopueringer/CorridorKey2.0) by Corridor Digital, designed and trained by Zachary Harris with Niko Pueringer. The v2 runtime in `corridorkey/` and `third_party/MoGe/` is vendored unmodified from it. +- [CorridorKey](https://github.com/nikopueringer/CorridorKey) v1 by Niko Pueringer / Corridor Digital (synced to `8a4e4b4`, 2026-05-01) - [EZ-CorridorKey](https://github.com/edenaion/EZ-CorridorKey) UI reference by edenaion (synced to `888e032`, 2026-04-25) - [BiRefNet](https://github.com/ZhengPeng7/BiRefNet) by ZhengPeng7 +- [C-RADIO](https://github.com/NVlabs/RADIO) (NVIDIA), [MoGe](https://github.com/microsoft/MoGe) (Microsoft), [RVM](https://github.com/google-deepmind/representations4d) (Google DeepMind) diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md new file mode 100644 index 0000000000000000000000000000000000000000..924012b75ffcf87f10880e152dd70a1d7b9dd730 --- /dev/null +++ b/THIRD_PARTY_NOTICES.md @@ -0,0 +1,39 @@ +# Third-Party Notices + +CorridorKey v2 is licensed under the Corridor Key Licence (see `LICENSE`). +It depends on the third-party components below, which are not owned by +Corridor Digital LLC and remain under their own licenses. + +The three foundation models are not redistributed in this repository or in the +CorridorKey Hugging Face repository. `scripts/download_weights.py` fetches them +from their original publishers. + +## C-RADIOv4-SO400M + +- Publisher: NVIDIA Corporation +- Source: https://huggingface.co/nvidia/C-RADIOv4-SO400M +- License: [NVIDIA Open Model License](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf) +- Use: frozen visual feature extractor, downloaded to `vfms/cradio/` + +Licensed by NVIDIA Corporation under the NVIDIA Open Model License. + +## MoGe-2 ViT-B + +- Publisher: Microsoft Corporation +- Source: https://github.com/microsoft/MoGe, weights at https://huggingface.co/Ruicheng/moge-2-vitb-normal +- License: MIT (code and weights) +- Use: frozen geometry feature extractor. The model code is vendored in + `third_party/MoGe/` with its license at `third_party/MoGe/LICENSE`; weights + are downloaded to `vfms/moge/`. + +The vendored MoGe code includes DINOv2 (Meta Platforms, Inc.), licensed under +the Apache License 2.0, also reproduced in `third_party/MoGe/LICENSE`. + +## RVM (Recurrent Video MAE), small + +- Publisher: Google DeepMind +- Source: https://github.com/google-deepmind/representations4d +- License: Apache License 2.0 (code), Creative Commons Attribution 4.0 (weights) +- Use: frozen recurrent video feature extractor. `corridorkey/rvm.py` is a + PyTorch port of the reference architecture; weights are downloaded to + `vfms/rvm/`. diff --git a/app.py b/app.py index feffa617f5c049b122ad38ef3ff6baf994bd1fb5..ee528411fb11c258550e922b340c20aa8020c5f5 100644 --- a/app.py +++ b/app.py @@ -24,6 +24,9 @@ import logging os.environ["OMP_NUM_THREADS"] = "2" os.environ["OPENBLAS_NUM_THREADS"] = "2" os.environ["MKL_NUM_THREADS"] = "2" +# Bound HF downloads so a stalled connection fails and retries instead of hanging startup. +os.environ.setdefault("HF_HUB_DOWNLOAD_TIMEOUT", "60") +os.environ.setdefault("HF_HUB_ETAG_TIMEOUT", "30") import numpy as np import cv2 @@ -104,6 +107,26 @@ except Exception: except Exception as e: logger.warning("CorridorKey green preload failed: %s", e) +# CorridorKey v2 (GPU only). v1 keeps working if v2 cannot load. +import ck2 + + +def _v1_cli_requested() -> bool: + """A CLI run that asks for v1 explicitly must not pay for the v2 download.""" + argv = sys.argv[1:] + if "--input" not in argv: + return False + for i, arg in enumerate(argv): + if arg == "--version" and i + 1 < len(argv): + return argv[i + 1] == "v1" + if arg.startswith("--version="): + return arg.split("=", 1)[1] == "v1" + return False + + +if not _v1_cli_requested(): + ck2.init() + # Batch sizes for GPU inference (conservative for H200 80GB) GPU_BATCH_SIZES = {"1024": 32, "2048": 16} # 2048 uses only 5.7GB/batch=2, so 16 easily fits in 69.8GB @@ -1014,9 +1037,47 @@ def process_video(video_path, resolution, despill_val, mask_mode, def process_example(video_path, resolution, despill, mask_mode, despeckle, despeckle_size, screen_color="Auto"): return process_video(video_path, resolution, despill, mask_mode, despeckle, despeckle_size, screen_color) -DESCRIPTION = """# CorridorKey Green/Blue Screen Matting -Remove green or blue screen backgrounds from video. Based on [CorridorKey_v1.0](https://www.youtube.com/watch?v=3Ploi723hg4) by Corridor Digital. -ZeroGPU H200: batched PyTorch inference (up to 32 frames at once). CPU fallback via ONNX (blue limited to 1024 on CPU).""" + +def process_video_v2(video_path: str, quality: str = ck2.DEFAULT_QUALITY, + temporal: str = ck2.DEFAULT_TEMPORAL, despill_val: int = 5, + upscale: bool = False, progress=gr.Progress()): + """CorridorKey v2 green/blue screen matting (GPU only). + + Args: + video_path: Green or blue screen video. + quality: Working resolution as short side, "540p", "720p" or "1080p". + temporal: Frames of temporal context, "auto", "1", "8" or "24". + despill_val: Despill strength 0-10 for the composite preview. + upscale: Upscale the previews to the source resolution. + + Returns: + Composite preview, alpha matte preview, full-package zip, status text. + """ + try: + return ck2.process(video_path, quality, temporal, despill_val, upscale, progress) + except gr.Error: + raise + except RuntimeError as e: + raise gr.Error(str(e)) + except Exception as e: + logger.exception("v2 failed") + raise gr.Error(f"CorridorKey v2 failed: {e}") + + +def process_example_v2(video_path, quality, temporal, despill, upscale): + return process_video_v2(video_path, quality, temporal, despill, upscale) + + +V2_OK = ck2.is_ready() +VERSION_CHOICES = ([("v2 (GPU, newer)", "v2")] if V2_OK else []) + [("v1 (GPU or CPU)", "v1")] +DEFAULT_VERSION = "v2" if V2_OK else "v1" + +DESCRIPTION = ( + f"CorridorKey | v2 (default): GPU only, about {ck2.seconds_per_frame(1280, 720):.2f}s per frame at 720p, " + f"up to {ck2.max_frames(1280, 720)} frames at 720p | v1: GPU or CPU (ONNX) at a fixed model resolution" +) +if not V2_OK: + DESCRIPTION += f" | v2 unavailable: {(ck2.load_error() or 'not loaded')[:120]}" with gr.Blocks(title="CorridorKey") as demo: gr.Markdown(DESCRIPTION) @@ -1024,53 +1085,104 @@ with gr.Blocks(title="CorridorKey") as demo: with gr.Row(): with gr.Column(scale=1): input_video = gr.Video(label="Upload Green/Blue Screen Video") + version = gr.Radio( + choices=VERSION_CHOICES, value=DEFAULT_VERSION, label="Model version", + info="v2: temporal model, GPU only. v1: single frame, also runs on CPU" + ) with gr.Accordion("Settings", open=True): - with gr.Row(): - resolution = gr.Radio( - choices=["1024", "2048"], value="1024", - label="Resolution", info="1024 = fast, 2048 = quality" - ) - screen_color = gr.Radio( - choices=["Auto", "Green", "Blue"], value="Auto", - label="Screen Color", info="Auto-detects. Blue requires GPU" - ) - mask_mode = gr.Radio( - choices=["Hybrid (auto)", "AI (BiRefNet)", "Fast (classical)"], - value="Hybrid (auto)", label="Mask Mode", - info="Hybrid = fast detection + AI fallback. Fast = classical only. AI = always BiRefNet" - ) despill_slider = gr.Slider( 0, 10, value=5, step=1, label="Despill Strength", info="Remove screen reflections (0=off, 10=max)" ) - with gr.Row(): - despeckle_check = gr.Checkbox(value=True, label="Auto Despeckle") - despeckle_size = gr.Number(value=400, precision=0, label="Min Area", info="Pixels to keep") - process_btn = gr.Button("Process Video", variant="primary", size="lg") + with gr.Column(visible=DEFAULT_VERSION == "v2") as v2_controls: + with gr.Row(): + quality = gr.Radio( + choices=ck2.QUALITY_CHOICES, value=ck2.DEFAULT_QUALITY, + label="Quality", info="Working resolution (short side). Higher is slower" + ) + temporal = gr.Radio( + choices=ck2.TEMPORAL_CHOICES, value=ck2.DEFAULT_TEMPORAL, + label="Temporal context", info="Frames seen together. Auto picks the widest cheap window" + ) + upscale_check = gr.Checkbox(value=False, label="Upscale previews to source resolution") + with gr.Column(visible=DEFAULT_VERSION == "v1") as v1_controls: + with gr.Row(): + resolution = gr.Radio( + choices=["1024", "2048"], value="1024", + label="Resolution", info="1024 = fast, 2048 = quality" + ) + screen_color = gr.Radio( + choices=["Auto", "Green", "Blue"], value="Auto", + label="Screen Color", info="Auto-detects. Blue requires GPU" + ) + mask_mode = gr.Radio( + choices=["Hybrid (auto)", "AI (BiRefNet)", "Fast (classical)"], + value="Hybrid (auto)", label="Mask Mode", + info="Hybrid = fast detection + AI fallback. Fast = classical only. AI = always BiRefNet" + ) + with gr.Row(): + despeckle_check = gr.Checkbox(value=True, label="Auto Despeckle") + despeckle_size = gr.Number(value=400, precision=0, label="Min Area", info="Pixels to keep") + process_btn_v2 = gr.Button("Process Video", variant="primary", size="lg", + visible=DEFAULT_VERSION == "v2") + process_btn = gr.Button("Process Video", variant="primary", size="lg", + visible=DEFAULT_VERSION == "v1") with gr.Column(scale=1): with gr.Row(): comp_video = gr.Video(label="Composite Preview") matte_video = gr.Video(label="Alpha Matte") - download_zip = gr.File(label="Download Full Package (Comp + FG + Matte + Processed)") + download_zip = gr.File(label="Download Full Package") status_text = gr.Textbox(label="Status", interactive=False) - gr.Examples( - examples=[ - ["examples/corridor_greenscreen_demo.mp4", "1024", 5, "Hybrid (auto)", True, 400, "Auto"], - ], - inputs=[input_video, resolution, despill_slider, mask_mode, despeckle_check, despeckle_size, screen_color], - outputs=[comp_video, matte_video, download_zip, status_text], - fn=process_example, - cache_examples=True, - cache_mode="lazy", - label="Examples (click to load)" + _outputs = [comp_video, matte_video, download_zip, status_text] + + with gr.Column(visible=DEFAULT_VERSION == "v2") as v2_examples: + gr.Examples( + examples=[["examples/corridor_greenscreen_demo.mp4", "720p", "auto", 5, False]], + inputs=[input_video, quality, temporal, despill_slider, upscale_check], + outputs=_outputs, + fn=process_example_v2, + cache_examples=True, + cache_mode="lazy", + label="Examples v2 (click to load)" + ) + with gr.Column(visible=DEFAULT_VERSION == "v1") as v1_examples: + gr.Examples( + examples=[ + ["examples/corridor_greenscreen_demo.mp4", "1024", 5, "Hybrid (auto)", True, 400, "Auto"], + ], + inputs=[input_video, resolution, despill_slider, mask_mode, despeckle_check, despeckle_size, screen_color], + outputs=_outputs, + fn=process_example, + cache_examples=True, + cache_mode="lazy", + label="Examples v1 (click to load)" + ) + + def _on_version(value): + is_v2 = value == "v2" + return (gr.update(visible=is_v2), gr.update(visible=not is_v2), + gr.update(visible=is_v2), gr.update(visible=not is_v2), + gr.update(visible=is_v2), gr.update(visible=not is_v2)) + + version.change( + _on_version, inputs=[version], + outputs=[v2_controls, v1_controls, process_btn_v2, process_btn, v2_examples, v1_examples], + api_name=False, ) process_btn.click( fn=process_video, inputs=[input_video, resolution, despill_slider, mask_mode, despeckle_check, despeckle_size, screen_color], - outputs=[comp_video, matte_video, download_zip, status_text], + outputs=_outputs, + api_name="process_video", + ) + process_btn_v2.click( + fn=process_video_v2, + inputs=[input_video, quality, temporal, despill_slider, upscale_check], + outputs=_outputs, + api_name="process_video_v2", ) @@ -1079,35 +1191,72 @@ with gr.Blocks(title="CorridorKey") as demo: # --------------------------------------------------------------------------- def cli_main(): import argparse - parser = argparse.ArgumentParser(description="CorridorKey Green/Blue Screen Matting") + parser = argparse.ArgumentParser( + description="CorridorKey Green/Blue Screen Matting. v2 by default; add --version v1 for v1.") parser.add_argument("--input", required=True) parser.add_argument("--output", default="output") - parser.add_argument("--device", default="auto", choices=["auto", "cpu", "cuda"]) - parser.add_argument("--resolution", default="1024", choices=["1024", "2048"]) - parser.add_argument("--mask-mode", default="Hybrid (auto)", - choices=["Hybrid (auto)", "AI (BiRefNet)", "Fast (classical)"]) + parser.add_argument("--version", default="v2", choices=["v2", "v1"], + help="model version (default: v2, GPU only)") parser.add_argument("--despill", type=int, default=5) - parser.add_argument("--no-despeckle", action="store_true") - parser.add_argument("--despeckle-size", type=int, default=400) - parser.add_argument("--screen-color", default="Auto", choices=["Auto", "Green", "Blue"]) + g1 = parser.add_argument_group("v1 only (needs --version v1)") + g1.add_argument("--device", default=None, choices=["auto", "cpu", "cuda"]) + g1.add_argument("--resolution", default=None, choices=["1024", "2048"]) + g1.add_argument("--mask-mode", default=None, + choices=["Hybrid (auto)", "AI (BiRefNet)", "Fast (classical)"]) + g1.add_argument("--no-despeckle", action="store_true") + g1.add_argument("--despeckle-size", type=int, default=None) + g1.add_argument("--screen-color", default=None, choices=["Auto", "Green", "Blue"]) + g2 = parser.add_argument_group("v2 only") + g2.add_argument("--quality", default=None, choices=ck2.QUALITY_CHOICES) + g2.add_argument("--temporal", default=None, choices=[v for _, v in ck2.TEMPORAL_CHOICES]) + g2.add_argument("--upscale", action="store_true", help="upscale previews to the source resolution") args = parser.parse_args() - global HAS_CUDA - if args.device == "cpu": HAS_CUDA = False - elif args.device == "cuda": HAS_CUDA = True - print(f"Device: {'CUDA' if HAS_CUDA else 'CPU'}") + v1_given = [flag for flag, value in ( + ("--device", args.device), ("--resolution", args.resolution), ("--mask-mode", args.mask_mode), + ("--no-despeckle", args.no_despeckle or None), ("--despeckle-size", args.despeckle_size), + ("--screen-color", args.screen_color)) if value is not None] + v2_given = [flag for flag, value in ( + ("--quality", args.quality), ("--temporal", args.temporal), + ("--upscale", args.upscale or None)) if value is not None] + if args.version == "v2" and v1_given: + parser.error(f"{', '.join(v1_given)}: v1-only option(s). Add --version v1 " + "(v2 needs a CUDA GPU and has no CPU mode).") + if args.version == "v1" and v2_given: + parser.error(f"{', '.join(v2_given)}: v2-only option(s). Drop them or use --version v2.") class CLIProgress: def __call__(self, val, desc=""): if desc: print(f" [{val:.0%}] {desc}") + os.makedirs(args.output, exist_ok=True) + if args.version == "v2": + if not ck2.is_ready(): + parser.exit(1, f"v2 is unavailable: {ck2.load_error() or 'not loaded'}\n" + "Use --version v1 for the v1 pipeline.\n") + comp, matte, zipf, status = ck2.process( + args.input, args.quality or ck2.DEFAULT_QUALITY, args.temporal or ck2.DEFAULT_TEMPORAL, + args.despill, args.upscale, progress=CLIProgress()) + print(f"\n{status}") + for path in (comp, matte, zipf): + if path: + dst = os.path.join(args.output, os.path.basename(path)) + shutil.copy2(path, dst) + print(f"Output: {dst}") + return + + global HAS_CUDA + device = args.device or "auto" + if device == "cpu": HAS_CUDA = False + elif device == "cuda": HAS_CUDA = True + print(f"Device: {'CUDA' if HAS_CUDA else 'CPU'}") + comp, matte, zipf, status = process_video( - args.input, args.resolution, args.despill, args.mask_mode, - not args.no_despeckle, args.despeckle_size, args.screen_color, - progress=CLIProgress() + args.input, args.resolution or "1024", args.despill, args.mask_mode or "Hybrid (auto)", + not args.no_despeckle, args.despeckle_size if args.despeckle_size is not None else 400, + args.screen_color or "Auto", progress=CLIProgress() ) print(f"\n{status}") - os.makedirs(args.output, exist_ok=True) if zipf: dst = os.path.join(args.output, os.path.basename(zipf)) shutil.copy2(zipf, dst) @@ -1119,4 +1268,4 @@ if __name__ == "__main__": cli_main() else: demo.queue(default_concurrency_limit=1) - demo.launch(ssr_mode=False, mcp_server=True) + demo.launch(ssr_mode=False, mcp_server=True, show_error=True) diff --git a/ck2.py b/ck2.py new file mode 100644 index 0000000000000000000000000000000000000000..ac67b8de2e8142037ded17f131dc5859b7bb2ba2 --- /dev/null +++ b/ck2.py @@ -0,0 +1,337 @@ +"""CorridorKey v2 adapter: assets, CPU-resident model, CPU pre/post, one GPU call. + +The vendored `corridorkey` package is the unmodified v2 runtime. This module only +decides what runs where, so the time spent inside the ZeroGPU window is model work: + + CPU probe + downscale the clip with ffmpeg + GPU run_inference (RVM prepass, Hann stage 1, stage 2, ffmpeg writers) + CPU browser-safe previews + zip + +v2 needs CUDA. There is no CPU path; v1 in app.py keeps that role. +""" + +from __future__ import annotations + +import base64 +import hashlib +import logging +import math +import os +import shutil +import subprocess +import sys +import tempfile +import time +import urllib.request +import zipfile +from pathlib import Path + +import cv2 + +try: + import spaces + HAS_SPACES = True +except ImportError: + HAS_SPACES = False + +logger = logging.getLogger("ck2") + +ROOT = Path(__file__).resolve().parent +if str(ROOT) not in sys.path: + sys.path.insert(0, str(ROOT)) +# Weights and foundation models live here. CK2_ASSET_ROOT lets a local run reuse an existing download. +ASSETS = Path(os.environ.get("CK2_ASSET_ROOT") or ROOT) + +CK2_REPO = "nikopueringer/CorridorKey2.0" +CK2_FILES = ("corridorkey_v2.safetensors", "foundation_stats.safetensors") +CRADIO_REPO = "nvidia/C-RADIOv4-SO400M" +CRADIO_REVISION = "c0457f5dc26ca145f954cd4fc5bb6114e5705ad8" +MOGE_REPO = "Ruicheng/moge-2-vitb-normal" +MOGE_REVISION = "ca5f0e07ff01d3e5a364c1d954ed12ee1814b368" +RVM_FILE = "pretrain_rvm_small16_256_204031069.npz" +RVM_URL = f"https://storage.googleapis.com/representations4d/checkpoints/{RVM_FILE}" +RVM_MD5_B64 = "ZJb6kj8+tasiJbVeplXu7Q==" + +QUALITY_CHOICES = ["540p", "720p", "1080p"] +DEFAULT_QUALITY = "720p" +TEMPORAL_CHOICES = [("Auto", "auto"), ("Off (1 frame)", "1"), ("8 frames", "8"), ("24 frames", "24")] +DEFAULT_TEMPORAL = "auto" +STAGE2_BATCH = 4 + +MAX_GPU_SECONDS = int(os.environ.get("CK2_MAX_GPU_SECONDS", "120")) +GPU_OVERHEAD_S = 25.0 +GPU_SAFETY = 1.35 + +_model = None +_load_error = "" +_ready = False + + +def seconds_per_frame(width: int, height: int) -> float: + """Measured on an RTX 5090 (89 frames, 540p/720p/1080p): about 0.05 + 0.22 s per megapixel.""" + return 0.05 + 0.22 * (width * height / 1e6) + + +def estimate_gpu_seconds(frames: int, width: int, height: int) -> int: + estimate = GPU_OVERHEAD_S + GPU_SAFETY * frames * seconds_per_frame(width, height) + return int(max(30, min(MAX_GPU_SECONDS, math.ceil(estimate)))) + + +def max_frames(width: int, height: int) -> int: + return max(1, int((MAX_GPU_SECONDS - GPU_OVERHEAD_S) / (GPU_SAFETY * seconds_per_frame(width, height)))) + + +def working_size(src_w: int, src_h: int, quality: str) -> tuple[int, int]: + """Largest size not above the quality's short side, even dimensions, source aspect kept.""" + target = int(quality.rstrip("p")) + short = min(src_w, src_h) + if short <= target: + return src_w - src_w % 2, src_h - src_h % 2 + scale = target / short + return max(2, round(src_w * scale / 2) * 2), max(2, round(src_h * scale / 2) * 2) + + +def resolve_window(choice: str, width: int, height: int) -> tuple[int, int]: + """Hann window (chunk, stride). Auto takes the widest window that stays close to the cost of 8 + frames: 80 up to 720p, 24 above (an 80-frame window at 1080p is about 2.5x slower).""" + if str(choice) == "auto": + chunk = 80 if width * height <= 1_000_000 else 24 + elif str(choice) in ("1", "8", "24"): + chunk = int(choice) + else: + raise RuntimeError(f"Unsupported temporal context {choice!r}; choose auto, 1, 8 or 24.") + return chunk, max(1, chunk // 2) + + +def _md5_b64(path: Path) -> str: + digest = hashlib.md5() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + digest.update(chunk) + return base64.b64encode(digest.digest()).decode() + + +def _retry(label: str, fn, attempts: int = 3): + for attempt in range(1, attempts + 1): + try: + return fn() + except Exception as exc: + logger.warning("[ck2] %s failed (%d/%d): %s", label, attempt, attempts, exc) + if attempt == attempts: + raise + time.sleep(3 * attempt) + + +def ensure_assets() -> None: + """Fetch the v2 weights and the three frozen foundation models, skipping anything present.""" + from huggingface_hub import hf_hub_download, snapshot_download + + for name in CK2_FILES: + _retry(name, lambda n=name: hf_hub_download(CK2_REPO, n, local_dir=ASSETS / "weights")) + cradio = ASSETS / "vfms/cradio/C-RADIOv4-SO400M" + _retry("C-RADIO", lambda: snapshot_download( + CRADIO_REPO, revision=CRADIO_REVISION, local_dir=cradio, ignore_patterns=["*.pth.tar"])) + _retry("MoGe", lambda: hf_hub_download( + MOGE_REPO, "model.pt", revision=MOGE_REVISION, local_dir=ASSETS / "vfms/moge")) + + dst = ASSETS / "vfms/rvm" / RVM_FILE + if dst.is_file() and _md5_b64(dst) == RVM_MD5_B64: + return + dst.parent.mkdir(parents=True, exist_ok=True) + tmp = dst.with_suffix(dst.suffix + ".part") + + def fetch(): + with urllib.request.urlopen(RVM_URL, timeout=60) as response, open(tmp, "wb") as f: + shutil.copyfileobj(response, f, length=1 << 20) + if _md5_b64(tmp) != RVM_MD5_B64: + tmp.unlink() + raise RuntimeError("checksum mismatch downloading RVM weights") + tmp.replace(dst) + + _retry("RVM", fetch) + + +def init() -> bool: + """Download assets and build the model on CPU. Never raises: v1 must keep working.""" + global _model, _ready, _load_error + try: + t0 = time.time() + ensure_assets() + t1 = time.time() + saved = {k: os.environ.get(k) for k in ("HF_HUB_OFFLINE", "TRANSFORMERS_OFFLINE")} + try: + from corridorkey import runtime as rt + _model = rt.load_model(rt.RuntimePaths.from_package(ASSETS), device="cpu") + finally: + for key, value in saved.items(): + if value is None: + os.environ.pop(key, None) + else: + os.environ[key] = value + _ready = True + logger.info("[ck2] ready: assets %.0fs, model load %.0fs", t1 - t0, time.time() - t1) + except Exception as exc: + _load_error = f"{type(exc).__name__}: {exc}" + logger.exception("[ck2] v2 unavailable") + return _ready + + +def is_ready() -> bool: + return _ready + + +def load_error() -> str: + return _load_error + + +def _run(cmd: list[str], what: str, timeout: int = 600) -> None: + try: + subprocess.run(cmd, capture_output=True, timeout=timeout, check=True) + except subprocess.CalledProcessError as exc: + raise RuntimeError(f"{what} failed: {exc.stderr.decode(errors='ignore')[-300:]}") from exc + except (FileNotFoundError, subprocess.TimeoutExpired) as exc: + raise RuntimeError(f"{what} failed: {exc}") from exc + + +def _probe(path: str) -> tuple[int, int, float, int]: + cap = cv2.VideoCapture(path) + try: + w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) + h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) + fps = cap.get(cv2.CAP_PROP_FPS) or 30.0 + frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) + finally: + cap.release() + if frames <= 0 or w <= 0 or h <= 0: + raise RuntimeError("Could not read video frames.") + return w, h, fps, frames + + +def prepare_input(video_path: str, quality: str, workdir: str) -> dict: + """CPU stage: validate against the GPU budget and downscale the clip once.""" + src_w, src_h, fps, frames = _probe(video_path) + w, h = working_size(src_w, src_h, quality) + limit = max_frames(w, h) + if frames > limit: + raise RuntimeError( + f"Video has {frames} frames ({frames / fps:.1f}s); at {quality} the limit is " + f"{limit} frames (~{limit / fps:.1f}s). Pick a lower quality or a shorter clip.") + work = os.path.join(workdir, "clip.mp4") + if (w, h) == (src_w, src_h): + cmd = ["ffmpeg", "-y", "-i", video_path, "-an", "-c:v", "libx264", "-preset", "veryfast", + "-crf", "8", "-pix_fmt", "yuv420p", work] + else: + cmd = ["ffmpeg", "-y", "-i", video_path, "-an", "-vf", f"scale={w}:{h}:flags=lanczos", + "-c:v", "libx264", "-preset", "veryfast", "-crf", "8", "-pix_fmt", "yuv420p", work] + _run(cmd, "Downscale") + return {"work": work, "w": w, "h": h, "src_w": src_w, "src_h": src_h, "fps": fps, "frames": frames} + + +def _gpu_duration(work_video, out_dir, frames, width, height, *args, **kwargs): + return estimate_gpu_seconds(int(frames), int(width), int(height)) + + +def _gpu_decorator(fn): + if HAS_SPACES: + return spaces.GPU(duration=_gpu_duration)(fn) + return fn + + +@_gpu_decorator +def _gpu_phase(work_video, out_dir, frames, width, height, chunk, stride, despill_strength): + """All GPU work. The model is moved in and moved back out so every call starts clean.""" + import torch + from corridorkey import runtime as rt + + if not torch.cuda.is_available(): + raise RuntimeError("CorridorKey v2 requires a CUDA GPU.") + t0 = time.time() + props = torch.cuda.get_device_properties(0) + gpu = f"{props.name} {props.total_memory / 2**30:.0f}GB" + tmp = Path(tempfile.mkdtemp(prefix="ck2_tmp_")) + try: + _model.to("cuda") + cfg = rt.InferenceConfig( + hann_chunk=int(chunk), hann_stride=int(stride), auto_vram=False, + stage2_batch=STAGE2_BATCH, outputs={"alpha", "fg", "checker"}, + despill_strength=float(despill_strength), temp_dir=tmp) + writer = rt.WriterConfig(preset="veryfast", threads=4) + rt.run_inference(_model, work_video, out_dir, cfg, writer, + initial_hint_path=None, hint_video_path=None) + finally: + _model.to("cpu") + torch.cuda.empty_cache() + shutil.rmtree(tmp, ignore_errors=True) + return time.time() - t0, gpu + + +def _find(out_dir: str, suffix: str) -> str | None: + for name in sorted(os.listdir(out_dir)): + if name.endswith(".mp4") and suffix in name: + return os.path.join(out_dir, name) + return None + + +def _preview(src: str, dst: str, size: tuple[int, int] | None) -> str: + vf = "format=yuv420p" if size is None else f"scale={size[0]}:{size[1]}:flags=lanczos,format=yuv420p" + _run(["ffmpeg", "-y", "-i", src, "-vf", vf, "-c:v", "libx264", "-preset", "fast", "-crf", "18", + "-movflags", "+faststart", dst], "Preview encode") + return dst + + +def process(video_path, quality=DEFAULT_QUALITY, temporal=DEFAULT_TEMPORAL, despill_val=5, + upscale=False, progress=None): + """Run CorridorKey v2. Returns (composite, matte, zip, status) like the v1 pipeline.""" + if not _ready: + raise RuntimeError(f"CorridorKey v2 is not available ({_load_error or 'still loading'}). " + "Switch to v1 or retry in a minute.") + if video_path is None: + raise RuntimeError("Please upload a video.") + if quality not in QUALITY_CHOICES: + raise RuntimeError(f"Unsupported quality {quality!r}; choose one of {QUALITY_CHOICES}.") + def step(frac, text): + if progress is not None: + progress(frac, desc=text) + + t_total = time.time() + workdir = tempfile.mkdtemp(prefix="ck2_job_") + out_dir = os.path.join(workdir, "out") + os.makedirs(out_dir) + + step(0.05, "Preparing clip (CPU)...") + t0 = time.time() + info = prepare_input(video_path, quality, workdir) + t_prep = time.time() - t0 + + chunk, stride = resolve_window(temporal, info["w"], info["h"]) + step(0.2, f"CorridorKey v2 on GPU ({info['w']}x{info['h']}, {info['frames']} frames)...") + t_gpu, gpu = _gpu_phase(info["work"], out_dir, info["frames"], info["w"], info["h"], + chunk, stride, float(despill_val) / 10.0) + + step(0.85, "Encoding previews (CPU)...") + t0 = time.time() + alpha = _find(out_dir, "_alpha") + fg = _find(out_dir, "_fg") + comp = _find(out_dir, "_checker") + if not (alpha and fg and comp): + raise RuntimeError("CorridorKey v2 finished without all expected outputs.") + size = None + if upscale and (info["src_w"], info["src_h"]) != (info["w"], info["h"]): + size = (info["src_w"] - info["src_w"] % 2, info["src_h"] - info["src_h"] % 2) + comp_preview = _preview(comp, os.path.join(workdir, "comp_preview.mp4"), size) + matte_preview = _preview(alpha, os.path.join(workdir, "matte_preview.mp4"), size) + zip_path = os.path.join(workdir, "CorridorKey2_Output.zip") + with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_STORED) as zf: + for path in (alpha, fg, comp): + zf.write(path, f"Output/{os.path.basename(path)}") + settings = [p for p in os.listdir(out_dir) if p.endswith("_settings.txt")] + for name in settings: + zf.write(os.path.join(out_dir, name), f"Output/{name}") + t_post = time.time() - t0 + + step(1.0, "Done!") + out_res = f"{size[0]}x{size[1]}" if size else f"{info['w']}x{info['h']}" + status = (f"CorridorKey v2 | {info['frames']} frames at {quality} ({out_res}) | " + f"temporal {chunk} | prep {t_prep:.0f}s, GPU {t_gpu:.0f}s ({gpu}), post {t_post:.0f}s, " + f"{time.time() - t_total:.0f}s total") + return comp_preview, matte_preview, zip_path, status diff --git a/corridorkey/__init__.py b/corridorkey/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..9bbcee91125b8d55ed0c96ae197e8ab04a6838d7 --- /dev/null +++ b/corridorkey/__init__.py @@ -0,0 +1,8 @@ +from .model import CorridorKeyV3 + + +class CorridorKeyV2(CorridorKeyV3): + """Public CorridorKey v2 runtime alias for the exported model.""" + + +__all__ = ["CorridorKeyV2"] diff --git a/corridorkey/blocks.py b/corridorkey/blocks.py new file mode 100644 index 0000000000000000000000000000000000000000..02841a98721500ceedf5659ac547f0bc6228aa1f --- /dev/null +++ b/corridorkey/blocks.py @@ -0,0 +1,3341 @@ +"""Building blocks for CorridorKey v2. + +Transformer-native matting stack. Spatial pyramid is /16 ↔ /64 only, +stage 2 runs pure /16 Swin-2D → depatchify. + +Publicly exported: + • LayerNorm2d — channel-LN on (B, C, H, W) tensors. + • SwiGLU — gated MLP, 2× expansion default (3 linears of d·2d each). + • SwinBlock3D — windowed 3D (T, H, W) attention + SwiGLU. + • FullSpatialBlock3D — per-frame full-spatial attention + SwiGLU. + • SwinBlock2D — windowed 2D attention + SwiGLU. + • FullSpatioTemporalBlock — unwindowed 3D attention (for /64 body). + • PatchifyConv — Conv(kernel=r, stride=r), optional masked-channel + zero-init (for stage-2 patchify_1 baton-overlap and patchify_2 skip). + • Downsample4 — kernel-8 stride-4 conv (for /16 → /64). + • Upsample4ICNR — Linear + PixelShuffle(4), ICNR-inited, optional + post-smoothing Conv(kernel=3). + • Depatchify — Linear(d → C·r²) + PixelShuffle(r) for /16 → /1 and /16 → /2. + • icnr_init_ — Aitken 2017 ICNR init for conv feeding PixelShuffle. + +Residual blocks use pre-norm plus zero-initialized output projections/gates so +every block is near-identity at t=0 and grows responsibility with training. +""" + +import math +from functools import lru_cache +from typing import Optional, Tuple + +import torch +import torch.nn as nn +import torch.nn.functional as F + +from .rope import RoPE2D, RoPE3D, apply_rotary + + +# --------------------------------------------------------------------------- +# Norms and utilities +# --------------------------------------------------------------------------- + + +class LayerNorm2d(nn.Module): + """RMSNorm over channel dim for (B, C, H, W) tensors. + + Kept the historical class name for backward compat of state-dict keys, + but the implementation is now RMSNorm: mean subtraction dropped, no + learnable bias, variance-only normalization. ~10-15% faster than + LayerNorm, and matches the EUPE / DINOv3 convention used by our + frozen foundations. + """ + + def __init__(self, num_channels: int, eps: float = 1e-6, + affine: bool = True): + super().__init__() + self.num_channels = num_channels + self.eps = eps + self.affine = affine + if affine: + self.weight = nn.Parameter(torch.ones(num_channels)) + else: + self.register_parameter('weight', None) + # RMSNorm has no bias; keep the attribute for any code that touches + # it defensively, but it is always None. + self.register_parameter('bias', None) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + # (B, C, H, W) → (B, H, W, C) → RMSNorm → back + x = x.permute(0, 2, 3, 1) + # Cast weight to input dtype so F.rms_norm's fused kernel dispatches + # (autocast keeps params fp32; mismatch blocks the fast path). + w = self.weight.to(x.dtype) if self.weight is not None else None + x = F.rms_norm(x, (self.num_channels,), w, self.eps) + return x.permute(0, 3, 1, 2).contiguous() + + +class RMSNorm(nn.Module): + """Drop-in replacement for nn.RMSNorm that casts `weight` to input dtype + inline so F.rms_norm's fused kernel dispatches under autocast. + + The stock nn.RMSNorm keeps `weight` in fp32; under autocast(bf16) the + activation is bf16, and F.rms_norm refuses the fused path on dtype + mismatch — this was the source of a perf-warning in training. + """ + def __init__(self, dim: int, eps: float = 1e-6): + super().__init__() + self.normalized_shape = (dim,) + self.weight = nn.Parameter(torch.ones(dim)) + self.eps = eps + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return F.rms_norm(x, self.normalized_shape, + self.weight.to(x.dtype), self.eps) + + +def icnr_init_(conv: nn.Conv2d, upscale_factor: int): + """ICNR init (Aitken 2017) for a Conv2d that feeds a PixelShuffle. + + Writes a "tile" where each r² group of output channels shares the + same sub-kernel — the initial PixelShuffle becomes a nearest-neighbor + upsample, so zero high-freq at t=0. + """ + r2 = upscale_factor * upscale_factor + W = conv.weight.data + out_ch, in_ch, kh, kw = W.shape + assert out_ch % r2 == 0, ( + f"icnr_init: out_channels ({out_ch}) must be divisible by r² ({r2}).") + c_out = out_ch // r2 + sub = torch.empty(c_out, in_ch, kh, kw, dtype=W.dtype, device=W.device) + nn.init.kaiming_uniform_(sub, a=5 ** 0.5) + W.copy_(sub.repeat_interleave(r2, dim=0)) + if conv.bias is not None: + conv.bias.data.zero_() + + +def icnr_init_linear_(linear: nn.Linear, upscale_factor: int, + out_channels_per_slot: int): + """ICNR init for a Linear feeding PixelShuffle. + + `linear.weight` has shape `(out_dim, in_dim)` with + `out_dim = out_channels_per_slot * upscale_factor²`. Groups of r² along + out_dim share the same slot-weight so PixelShuffle at init is nearest- + neighbor. + """ + r2 = upscale_factor * upscale_factor + out_dim, in_dim = linear.weight.data.shape + assert out_dim == out_channels_per_slot * r2, ( + f"icnr_init_linear: out_dim={out_dim} != c·r² = " + f"{out_channels_per_slot}·{r2}") + sub = torch.empty(out_channels_per_slot, in_dim, + dtype=linear.weight.dtype, device=linear.weight.device) + nn.init.kaiming_uniform_(sub, a=5 ** 0.5) + linear.weight.data.copy_(sub.repeat_interleave(r2, dim=0)) + if linear.bias is not None: + linear.bias.data.zero_() + + +def _random_orthogonal_matrix(dim: int, device: torch.device) -> torch.Tensor: + q, r = torch.linalg.qr(torch.randn(dim, dim, device=device)) + signs = torch.sign(torch.diagonal(r)) + signs = torch.where(signs == 0, torch.ones_like(signs), signs) + return q * signs.view(1, -1) + + +@torch.no_grad() +def init_orthogonal_rgb_passthrough_( + patchify: 'PatchifyConv', + depatchify: 'Depatchify', + *, + token_offset: int = 0, + output_rgb_offset: int = 1, + downsample: int = 1, +) -> bool: + """Initialize patchify/depatchify as an exact RGB passthrough. + + A full random orthogonal basis maps each native RGB patch to tokens when + the token width is 768. For lower-width stage-1 aux heads, the encoded + subspace is built to contain the exact area-downsampled RGB rows, plus a + random orthogonal complement. Returns False when the requested passthrough + cannot be represented exactly. + """ + if patchify.overlap_mult != 1 or depatchify.overlap_mult != 1: + return False + if depatchify.linear is None: + return False + + patch = patchify.patch + r = depatchify.r + if patch != r * downsample: + return False + + rgb_dim = 3 * patch * patch + target_dim = 3 * r * r + out_dim = patchify.conv.out_channels + if out_dim > rgb_dim: + return False + if out_dim < target_dim: + return False + if patchify.conv.in_channels < 3: + return False + if token_offset < 0 or token_offset + out_dim > depatchify.in_dim: + return False + if output_rgb_offset < 0 or output_rgb_offset + 3 > depatchify.out_channels: + return False + + basis_dtype = patchify.conv.weight.dtype + device = patchify.conv.weight.device + if out_dim == rgb_dim: + basis = _random_orthogonal_matrix(rgb_dim, device) + else: + target_rows = torch.zeros(target_dim, rgb_dim, device=device) + for c in range(3): + for oy in range(r): + for ox in range(r): + row = c * r * r + oy * r + ox + y0 = oy * downsample + x0 = ox * downsample + for dy in range(downsample): + for dx in range(downsample): + p_idx = c * patch * patch + (y0 + dy) * patch + (x0 + dx) + target_rows[row, p_idx] = 1.0 / downsample + rot = _random_orthogonal_matrix(target_dim, device) + fixed_rows = rot @ target_rows + rand = torch.randn(rgb_dim, out_dim - target_dim, device=device) + rand = rand - fixed_rows.T @ (fixed_rows @ rand) + q, _ = torch.linalg.qr(rand, mode='reduced') + basis = torch.cat([fixed_rows, q.T], dim=0) + basis = basis.to(dtype=basis_dtype) + + patchify.conv.weight.zero_() + patchify.conv.weight[:, :3, :patch, :patch].copy_( + basis.view(out_dim, 3, patch, patch) + ) + if patchify.conv.bias is not None: + patchify.conv.bias.zero_() + + depatchify.linear.weight.zero_() + if depatchify.linear.bias is not None: + depatchify.linear.bias.zero_() + if depatchify.fourier is not None: + depatchify.fourier.reset_to_zero() + w = depatchify.linear.weight + for c in range(3): + out_c = output_rgb_offset + c + for oy in range(r): + for ox in range(r): + row = out_c * r * r + oy * r + ox + cols = [] + y0 = oy * downsample + x0 = ox * downsample + for dy in range(downsample): + for dx in range(downsample): + p_idx = c * patch * patch + (y0 + dy) * patch + (x0 + dx) + cols.append(p_idx) + target = torch.zeros(rgb_dim, device=device, dtype=basis.dtype) + for p_idx in cols: + target[p_idx] = 1.0 / len(cols) + vec = basis @ target + w[row, token_offset:token_offset + out_dim].copy_(vec) + return True + + +# --------------------------------------------------------------------------- +# MLP: SwiGLU 2× expansion +# --------------------------------------------------------------------------- + + +class SwiGLU(nn.Module): + """Gated MLP (SwiGLU) — equal-dim variant (in == out == dim). + + `h = silu(W_gate @ x) * (W_up @ x); out = W_down @ h`. + Optional `dw_kernel` inserts a depthwise spatial conv on `h` before + `W_down`, i.e. post-gating and pre-down-projection. + At `mult=2.5` this is 7.5·dim² params (vs GELU-MLP 4× = 8·dim²) — smaller + than the param-matched GELU-4× SwiGLU (mult ≈ 2.67), intentional + VRAM/compute tradeoff. + + `zero_init_down=True` (default) zeros `w_down.weight` and `w_down.bias` + so the residual branch contributes exactly 0 at init, letting us drop + LayerScale gating. Set `zero_init_down=False` to keep Kaiming init + (legacy behavior that relied on LayerScale for near-identity). + """ + + def __init__( + self, + dim: int, + mult: float = 2.5, + bias: bool = True, + zero_init_down: bool = True, + dw_kernel: int = 0, + hidden: int | None = None, + ): + super().__init__() + hidden = int(hidden) if hidden is not None else int(round(dim * mult)) + self.hidden = hidden + self.dw_pad = int(dw_kernel) // 2 if dw_kernel and dw_kernel > 0 else 0 + self.w_gate = nn.Linear(dim, hidden, bias=bias) + self.w_up = nn.Linear(dim, hidden, bias=bias) + self.dw = ( + nn.Conv2d(hidden, hidden, kernel_size=int(dw_kernel), + padding=0, groups=hidden, bias=True) + if dw_kernel and dw_kernel > 0 else None + ) + self.w_down = nn.Linear(hidden, dim, bias=bias) + if zero_init_down: + nn.init.zeros_(self.w_down.weight) + if self.w_down.bias is not None: + nn.init.zeros_(self.w_down.bias) + + def _dw_spatial(self, h: torch.Tensor) -> torch.Tensor: + if self.dw is None: + return h + if h.dim() == 5: + B, T, H, W, C = h.shape + y = h.reshape(B * T, H, W, C) + shape_out = (B, T, H, W, C) + elif h.dim() == 4: + B, H, W, C = h.shape + y = h + shape_out = (B, H, W, C) + else: + return h + y = y.permute(0, 3, 1, 2) + if self.dw_pad: + mode = 'reflect' if H > self.dw_pad and W > self.dw_pad else 'replicate' + y = F.pad( + y, + (self.dw_pad, self.dw_pad, self.dw_pad, self.dw_pad), + mode=mode, + ) + y = self.dw(y).permute(0, 2, 3, 1) + return y.reshape(shape_out) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + chunk_tokens = int(getattr(self, 'inference_chunk_tokens', 0) or 0) + if (chunk_tokens > 0 and not self.training and self.dw is None + and x.numel() // x.shape[-1] > chunk_tokens): + shape = x.shape + c = shape[-1] + y = x.reshape(-1, c) + out = y.new_empty(y.shape) + for start in range(0, y.shape[0], chunk_tokens): + yi = y[start:start + chunk_tokens] + h = F.silu(self.w_gate(yi)) * self.w_up(yi) + out[start:start + yi.shape[0]] = self.w_down(h) + return out.reshape(shape) + h = F.silu(self.w_gate(x)) * self.w_up(x) + h = self._dw_spatial(h) + return self.w_down(h) + + +class DepthwiseSwiGLU2D(nn.Module): + """Depthwise spatial SwiGLU branch for NHWC feature maps. + + The branch is gated by a per-channel gamma initialized to exact zero, so it + is an exact residual identity at init. + The value and gate kernels are both depthwise 7x7 filters. After gamma + learns away from zero, they provide local translation-equivariant mixing + without adding pointwise channel mixing that the MLP already supplies. + """ + + def __init__( + self, + dim: int, + kernel_size: int = 7, + init_std: float | None = None, + ): + super().__init__() + assert kernel_size > 0 and kernel_size % 2 == 1, ( + f"kernel_size must be a positive odd integer; got {kernel_size}") + self.dim = dim + self.kernel_size = kernel_size + self.norm = RMSNorm(dim) + self.pad = kernel_size // 2 + self.value = nn.Conv2d(dim, dim, kernel_size=kernel_size, + padding=0, groups=dim, bias=True) + self.gate = nn.Conv2d(dim, dim, kernel_size=kernel_size, + padding=0, groups=dim, bias=True) + if init_std is None: + nn.init.kaiming_uniform_(self.value.weight, a=math.sqrt(5)) + nn.init.kaiming_uniform_(self.gate.weight, a=math.sqrt(5)) + else: + nn.init.normal_(self.value.weight, mean=0.0, std=init_std) + nn.init.normal_(self.gate.weight, mean=0.0, std=init_std) + nn.init.zeros_(self.value.bias) + nn.init.zeros_(self.gate.bias) + self.gamma = nn.Parameter(torch.zeros(dim)) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """x: (B, H, W, C). Returns a residual branch in the same layout.""" + u = self.norm(x) + # NHWC-contiguous -> NCHW shape with channels-last strides. Avoid an + # explicit contiguous copy; cuDNN handles this layout efficiently. + u = u.permute(0, 3, 1, 2) + if self.pad: + mode = 'reflect' if x.shape[1] > self.pad and x.shape[2] > self.pad else 'replicate' + u = F.pad(u, (self.pad, self.pad, self.pad, self.pad), + mode=mode) + # One grouped conv with two filters per input channel. Keep value/gate + # as separate parameters for optimizer state and checkpoint clarity, + # but avoid launching two depthwise convs over the same activation. + weight = torch.stack((self.value.weight, self.gate.weight), dim=1) + weight = weight.reshape(2 * self.dim, 1, self.kernel_size, self.kernel_size) + if self.value.bias is not None and self.gate.bias is not None: + bias = torch.stack((self.value.bias, self.gate.bias), dim=1).reshape(2 * self.dim) + else: + bias = None + out_elems_per_item = 2 * self.dim * x.shape[1] * x.shape[2] + max_items = max(1, 1_000_000_000 // max(1, out_elems_per_item)) + if u.shape[0] > max_items: + y = torch.cat([ + F.conv2d(u[i:i + max_items], weight, bias=bias, groups=self.dim) + for i in range(0, u.shape[0], max_items) + ], dim=0) + else: + y = F.conv2d(u, weight, bias=bias, groups=self.dim) + y = y.view(y.shape[0], self.dim, 2, y.shape[2], y.shape[3]) + y = y[:, :, 0] * F.silu(y[:, :, 1]) + y = y * self.gamma.to(y.dtype).view(1, -1, 1, 1) + return y.permute(0, 2, 3, 1) + + +class AdaLNZero(nn.Module): + """Zero-init global conditioning for NHWC/NTHWC pre-norm activations.""" + + def __init__(self, dim: int, cond_dim: int = 1, pairs: int = 2, + hidden_dim: int | None = None): + super().__init__() + self.dim = int(dim) + self.pairs = int(pairs) + hidden = int(hidden_dim or dim) + self.in_proj = nn.Linear(int(cond_dim), hidden, bias=True) + self.act = nn.SiLU() + self.proj = nn.Linear(hidden, self.pairs * 3 * dim, bias=True) + nn.init.zeros_(self.proj.weight) + nn.init.zeros_(self.proj.bias) + + def forward(self, cond: torch.Tensor, x: torch.Tensor) -> tuple[torch.Tensor, ...]: + if cond.dim() == 2: + cond = cond.unsqueeze(-1) + if cond.dim() != 3: + raise ValueError( + f"AdaLNZero cond must be (B,T) or (B,T,C), got {tuple(cond.shape)}") + cond = cond.to(device=x.device, dtype=self.in_proj.weight.dtype) + params = self.proj(self.act(self.in_proj(cond))) + params = params.to(dtype=x.dtype) + if x.dim() == 5: + view_shape = (x.shape[0], x.shape[1], 1, 1, self.dim) + elif x.dim() == 4: + view_shape = (x.shape[0], 1, 1, self.dim) + else: + raise ValueError(f"AdaLNZero x must be NHWC or NTHWC, got {tuple(x.shape)}") + return tuple(p.reshape(view_shape) for p in params.chunk(self.pairs * 3, dim=-1)) + + +class FusionLinSwiglu(nn.Module): + """Linear projection with a SwiGLU wrapped AROUND it — for /16 ↔ VFM + fusion at /64. + + Input: `(..., vfm_dim + aux_dim)` (the concat of VFM features and + auxiliary downsampled stage-1 features). + Output: `(..., out_dim)` (default: == vfm_dim). + + Two parallel paths, summed: + + • Straight linear shortcut: + Linear(vfm_dim + aux_dim → out_dim), initialized as + `[P | 0]`, where P is identity when dimensions match and otherwise + a variance-preserving semi-orthogonal projection over every VFM + channel. At t=0, the /64 stream is a stable VFM-derived latent and + the aux half contributes zero. + + • Nonlinear SwiGLU refinement: + LayerNorm → Linear(in → hidden) × 2 (gate + up) → silu(gate)·up + → Linear(hidden → out_dim). `hidden = mult · in` (default 2×). + The final Linear's weight/bias are zero-inited, so at t=0 this + path contributes 0. Since hidden > in > out, the SwiGLU UP- + projects the full input BEFORE any compression, giving + genuine nonlinear mixing between the vfm and aux channels + prior to collapse. + + Combined behavior: at t=0, `output == P(vfm)`. During training the + shortcut learns source weighting and the SwiGLU learns nonlinear + interactions between VFM and aux features. + """ + + def __init__(self, vfm_dim: int, aux_dim: int, out_dim: int = None, + mult: float = 2.5, bias: bool = True, + hidden: int = None): + """If `hidden` is None, it defaults to `round((vfm_dim + aux_dim) · mult)`. + Pass an explicit `hidden` to hold the SwiGLU hidden size constant + when retrofitting new input channels (e.g. adding RVM features) + without expanding the hidden layer — preserves the existing trained + hidden units.""" + super().__init__() + if out_dim is None: + out_dim = vfm_dim + in_dim = vfm_dim + aux_dim + self.vfm_dim = vfm_dim + self.aux_dim = aux_dim + self.in_dim = in_dim + self.out_dim = out_dim + if hidden is None: + hidden = int(round(in_dim * mult)) + self.hidden = hidden + + # Straight linear shortcut — semi-orthogonal VFM projection + zero aux. + self.linear_shortcut = nn.Linear(in_dim, out_dim, bias=bias) + with torch.no_grad(): + self.linear_shortcut.weight.zero_() + if vfm_dim > 0: + if out_dim == vfm_dim: + idx = torch.arange(out_dim) + self.linear_shortcut.weight[idx, idx] = 1.0 + elif out_dim < vfm_dim: + # Compress: rows are orthonormal, so whitened VFM features + # retain unit variance per output channel. + q, r = torch.linalg.qr( + torch.randn(vfm_dim, out_dim, + device=self.linear_shortcut.weight.device, + dtype=torch.float32), + mode='reduced', + ) + signs = torch.sign(torch.diagonal(r)) + signs = torch.where(signs == 0, torch.ones_like(signs), signs) + proj = (q * signs.view(1, -1)).T + self.linear_shortcut.weight[:, :vfm_dim].copy_( + proj.to(dtype=self.linear_shortcut.weight.dtype)) + else: + # Expand: columns are orthonormal, so the VFM vector norm is + # preserved while occupying the full output channel space. + q, r = torch.linalg.qr( + torch.randn(out_dim, vfm_dim, + device=self.linear_shortcut.weight.device, + dtype=torch.float32), + mode='reduced', + ) + signs = torch.sign(torch.diagonal(r)) + signs = torch.where(signs == 0, torch.ones_like(signs), signs) + proj = q * signs.view(1, -1) + self.linear_shortcut.weight[:, :vfm_dim].copy_( + proj.to(dtype=self.linear_shortcut.weight.dtype)) + if self.linear_shortcut.bias is not None: + self.linear_shortcut.bias.zero_() + + # SwiGLU path — own pre-norm; zero-init on down so contributes 0 at t=0. + self.norm = RMSNorm(in_dim) + self.w_gate = nn.Linear(in_dim, hidden, bias=bias) + self.w_up = nn.Linear(in_dim, hidden, bias=bias) + self.w_down = nn.Linear(hidden, out_dim, bias=bias) + nn.init.zeros_(self.w_down.weight) + if self.w_down.bias is not None: + nn.init.zeros_(self.w_down.bias) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """x: (..., in_dim). Returns (..., out_dim). At t=0, output is a + stable projection of the VFM portion of the input; aux/SwiGLU are zero.""" + chunk_tokens = int(getattr(self, 'inference_chunk_tokens', 0) or 0) + if (chunk_tokens > 0 and not self.training + and x.numel() // x.shape[-1] > chunk_tokens): + shape = x.shape[:-1] + (self.out_dim,) + y = x.reshape(-1, x.shape[-1]) + out = y.new_empty((y.shape[0], self.out_dim)) + for start in range(0, y.shape[0], chunk_tokens): + yi = y[start:start + chunk_tokens] + shortcut = self.linear_shortcut(yi) + h = self.norm(yi) + nonlinear = self.w_down(F.silu(self.w_gate(h)) * self.w_up(h)) + out[start:start + yi.shape[0]] = shortcut + nonlinear + return out.reshape(shape) + shortcut = self.linear_shortcut(x) + h = self.norm(x) + nonlinear = self.w_down(F.silu(self.w_gate(h)) * self.w_up(h)) + return shortcut + nonlinear + + +# --------------------------------------------------------------------------- +# Window partitioning (2D + 3D) +# --------------------------------------------------------------------------- + + +def window_partition_2d(x: torch.Tensor, window_size: int) -> Tuple[torch.Tensor, Tuple[int, int]]: + """(B, H, W, C) → (B·num_windows, window²·C flattened later) layout. + + Returns (windows, (H, W)) where windows is (B·nH·nW, window, window, C). + Callers use `windows.flatten(1, 2)` to get (B·nH·nW, window², C). + """ + B, H, W, C = x.shape + assert H % window_size == 0 and W % window_size == 0, ( + f"window_partition_2d: H×W ({H}×{W}) must divide window={window_size}") + x = x.view(B, H // window_size, window_size, + W // window_size, window_size, C) + windows = x.permute(0, 1, 3, 2, 4, 5).contiguous() + return windows.view(-1, window_size, window_size, C), (H, W) + + +def window_unpartition_2d(windows: torch.Tensor, window_size: int, + HW: Tuple[int, int], B: int) -> torch.Tensor: + """Inverse of window_partition_2d. Returns (B, H, W, C).""" + H, W = HW + x = windows.view(B, H // window_size, W // window_size, + window_size, window_size, -1) + x = x.permute(0, 1, 3, 2, 4, 5).contiguous() + return x.view(B, H, W, -1) + + +def window_partition_3d(x: torch.Tensor, T_w: int, H_w: int, W_w: int, + ) -> Tuple[torch.Tensor, Tuple[int, int, int]]: + """(B, T, H, W, C) → (B·num_windows, T_w·H_w·W_w, C) later flattened. + + Returns (windows, (T, H, W)) where windows is + (B·nT·nH·nW, T_w, H_w, W_w, C). + """ + B, T, H, W, C = x.shape + assert T % T_w == 0 and H % H_w == 0 and W % W_w == 0, ( + f"window_partition_3d: shape ({T},{H},{W}) vs window " + f"({T_w},{H_w},{W_w}) — must divide evenly.") + x = x.view(B, T // T_w, T_w, H // H_w, H_w, W // W_w, W_w, C) + # Reorder to (B, n_t, n_h, n_w, T_w, H_w, W_w, C) + windows = x.permute(0, 1, 3, 5, 2, 4, 6, 7).contiguous() + return windows.view(-1, T_w, H_w, W_w, C), (T, H, W) + + +def window_unpartition_3d(windows: torch.Tensor, T_w: int, H_w: int, W_w: int, + THW: Tuple[int, int, int], B: int) -> torch.Tensor: + """Inverse of window_partition_3d. Returns (B, T, H, W, C).""" + T, H, W = THW + x = windows.view(B, T // T_w, H // H_w, W // W_w, T_w, H_w, W_w, -1) + x = x.permute(0, 1, 4, 2, 5, 3, 6, 7).contiguous() + return x.view(B, T, H, W, -1) + + +def _shifted_region_ids_2d( + H: int, + W: int, + Hp: int, + Wp: int, + window_size: int, + shift_h: int, + shift_w: int, + device: torch.device, +) -> torch.Tensor | None: + """Per-window region ids for standard shifted-window masking. + + The ids are built on the padded pre-shift grid, then cyclically shifted + exactly like the activations and partitioned into windows. Tokens with + different non-negative ids must not attend to each other; id -1 denotes + padding. This represents the standard Swin wraparound mask without + materializing a full N×N mask, which is critical for full-temporal 3D + windows. + """ + + sh, sw = int(shift_h), int(shift_w) + if sh == 0 and sw == 0: + return None + w = int(window_size) + region = torch.full((Hp, Wp), -1, dtype=torch.int16, device=device) + real = torch.zeros((Hp, Wp), dtype=torch.bool, device=device) + real[:H, :W] = True + h_slices = ( + (slice(0, -w), slice(-w, -sh), slice(-sh, None)) + if sh > 0 else (slice(0, None),) + ) + w_slices = ( + (slice(0, -w), slice(-w, -sw), slice(-sw, None)) + if sw > 0 else (slice(0, None),) + ) + idx = 0 + for hs in h_slices: + for ws in w_slices: + sub = region[hs, ws] + sub_real = real[hs, ws] + sub[sub_real] = idx + region[hs, ws] = sub + idx += 1 + region = torch.roll(region, shifts=(-sh, -sw), dims=(0, 1)) + nH, nW = Hp // w, Wp // w + region = region.view(nH, w, nW, w).permute(0, 2, 1, 3).contiguous() + return region.view(nH * nW, w * w) + + +def _shifted_attention_mask_2d( + H: int, + W: int, + Hp: int, + Wp: int, + window_size: int, + shift_h: int, + shift_w: int, + device: torch.device, +) -> torch.Tensor | None: + """Standard Swin shifted-window bool mask for 2D SDPA. + + This is the high-throughput path used when the attention window is small + enough to materialize the per-window N x N mask. It also handles padded + tokens by masking padded keys from real queries; padded queries are cropped + after reverse shift, so they are allowed to attend to real keys only to + keep SDPA rows finite. + """ + + region = _shifted_region_ids_2d( + H, W, Hp, Wp, window_size, shift_h, shift_w, device) + if region is None: + return None + q_real = region.unsqueeze(2) >= 0 + k_real = region.unsqueeze(1) >= 0 + same_region = region.unsqueeze(1) == region.unsqueeze(2) + allowed = same_region & q_real & k_real + return torch.where(q_real, allowed, k_real.expand_as(allowed)) + + +def _shifted_region_ids_3d( + T: int, + H: int, + W: int, + Hp: int, + Wp: int, + window_H: int, + window_W: int, + shift_h: int, + shift_w: int, + device: torch.device, +) -> torch.Tensor | None: + # window_H == window_W for current Swin3D callers; keep the assertion + # explicit so a future rectangular-window change updates this helper. + if int(window_H) != int(window_W): + raise ValueError("shifted 3D region ids currently require square spatial windows") + spatial = _shifted_region_ids_2d( + H, W, Hp, Wp, window_H, shift_h, shift_w, device) + if spatial is None: + return None + return spatial.unsqueeze(1).expand(-1, T, -1).reshape(spatial.shape[0], T * spatial.shape[1]) + + +def _sdpa_with_region_ids( + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + region_ids: torch.Tensor | None, +) -> torch.Tensor: + """SDPA with standard shifted-window region isolation. + + `region_ids` has shape (num_windows, tokens). Tokens attend only to other + tokens in the same non-negative region. Windows that are a single region + are batched through normal SDPA; only wrapped/padded edge windows are split + into smaller SDPA calls. + """ + + if region_ids is None: + return F.scaled_dot_product_attention( + q, k, v, dropout_p=0.0, is_causal=False) + if region_ids.shape != (q.shape[0], q.shape[2]): + raise ValueError( + f"region_ids shape {tuple(region_ids.shape)} does not match " + f"attention windows/tokens {(q.shape[0], q.shape[2])}") + + out = torch.zeros_like(q) + valid = region_ids >= 0 + first = region_ids[:, :1] + full = valid.all(dim=1) & (region_ids == first).all(dim=1) + full_idx = torch.nonzero(full, as_tuple=False).flatten() + if full_idx.numel() > 0: + out.index_copy_( + 0, + full_idx, + F.scaled_dot_product_attention( + q.index_select(0, full_idx), + k.index_select(0, full_idx), + v.index_select(0, full_idx), + dropout_p=0.0, + is_causal=False, + ), + ) + + partial = torch.nonzero(~full, as_tuple=False).flatten() + if partial.numel() == 0: + return out + + # Shift masks have only a small number of repeated token-region layouts + # across edge windows. Batch identical token slices together instead of + # launching one SDPA call per window/region. + groups: dict[tuple[int, ...], list[int]] = {} + partial_cpu = region_ids.index_select(0, partial).detach().cpu() + partial_list = partial.detach().cpu().tolist() + for local_i, win_idx in enumerate(partial_list): + row = partial_cpu[local_i] + for label in torch.unique(row).tolist(): + if label < 0: + continue + token_tuple = tuple(torch.nonzero(row == label, as_tuple=False) + .flatten().tolist()) + if token_tuple: + groups.setdefault(token_tuple, []).append(int(win_idx)) + + out_nlhd = out.permute(0, 2, 1, 3) + for token_tuple, win_list in groups.items(): + win_idx = torch.tensor(win_list, device=q.device, dtype=torch.long) + token_idx = torch.tensor(token_tuple, device=q.device, dtype=torch.long) + attn = F.scaled_dot_product_attention( + q.index_select(0, win_idx).index_select(2, token_idx), + k.index_select(0, win_idx).index_select(2, token_idx), + v.index_select(0, win_idx).index_select(2, token_idx), + dropout_p=0.0, + is_causal=False, + ) + out_nlhd[win_idx[:, None], token_idx[None, :]] = attn.permute(0, 2, 1, 3) + return out + + +@lru_cache(maxsize=512) +def _shifted_region_groups_2d_cached( + H: int, + W: int, + Hp: int, + Wp: int, + window_size: int, + shift_h: int, + shift_w: int, +) -> tuple[int, tuple[int, ...], tuple[tuple[tuple[int, ...], tuple[int, ...]], ...]]: + region = _shifted_region_ids_2d( + H, W, Hp, Wp, window_size, shift_h, shift_w, torch.device('cpu')) + if region is None: + return 0, (), () + num_windows = int(region.shape[0]) + valid = region >= 0 + first = region[:, :1] + full = valid.all(dim=1) & (region == first).all(dim=1) + full_idx = tuple(torch.nonzero(full, as_tuple=False).flatten().tolist()) + + groups: dict[tuple[int, ...], list[int]] = {} + partial = torch.nonzero(~full, as_tuple=False).flatten().tolist() + for win_idx in partial: + row = region[win_idx] + for label in torch.unique(row).tolist(): + if label < 0: + continue + token_tuple = tuple(torch.nonzero(row == label, as_tuple=False) + .flatten().tolist()) + if token_tuple: + groups.setdefault(token_tuple, []).append(int(win_idx)) + return num_windows, full_idx, tuple( + (token_tuple, tuple(win_list)) + for token_tuple, win_list in groups.items() + ) + + +@lru_cache(maxsize=512) +def _shifted_region_groups_3d_cached( + T: int, + H: int, + W: int, + Hp: int, + Wp: int, + window_H: int, + window_W: int, + shift_h: int, + shift_w: int, +) -> tuple[int, tuple[int, ...], tuple[tuple[tuple[int, ...], tuple[int, ...]], ...]]: + if int(window_H) != int(window_W): + raise ValueError("shifted 3D region groups currently require square spatial windows") + spatial = _shifted_region_ids_2d( + H, W, Hp, Wp, window_H, shift_h, shift_w, torch.device('cpu')) + if spatial is None: + return 0, (), () + num_windows = int(spatial.shape[0]) + valid = spatial >= 0 + first = spatial[:, :1] + full = valid.all(dim=1) & (spatial == first).all(dim=1) + full_idx = tuple(torch.nonzero(full, as_tuple=False).flatten().tolist()) + + spatial_tokens = window_H * window_W + groups: dict[tuple[int, ...], list[int]] = {} + partial = torch.nonzero(~full, as_tuple=False).flatten().tolist() + for win_idx in partial: + row = spatial[win_idx] + for label in torch.unique(row).tolist(): + if label < 0: + continue + sp = torch.nonzero(row == label, as_tuple=False).flatten().tolist() + if not sp: + continue + token_tuple = tuple( + t * spatial_tokens + int(idx) + for t in range(int(T)) + for idx in sp + ) + groups.setdefault(token_tuple, []).append(int(win_idx)) + return num_windows, full_idx, tuple( + (token_tuple, tuple(win_list)) + for token_tuple, win_list in groups.items() + ) + + +def _sdpa_with_region_groups( + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + num_windows: int, + full_idx: tuple[int, ...], + groups: tuple[tuple[tuple[int, ...], tuple[int, ...]], ...], +) -> torch.Tensor: + if num_windows <= 0: + return F.scaled_dot_product_attention( + q, k, v, dropout_p=0.0, is_causal=False) + if q.shape[0] % num_windows != 0: + raise ValueError( + f"cached window count {num_windows} does not divide attention " + f"window batch {q.shape[0]}") + batch_windows = q.shape[0] // num_windows + + def _batched_windows(base: tuple[int, ...]) -> torch.Tensor: + idx = torch.tensor(base, device=q.device, dtype=torch.long) + if batch_windows == 1 or idx.numel() == 0: + return idx + offsets = torch.arange( + batch_windows, device=q.device, dtype=torch.long) * num_windows + return (idx.unsqueeze(0) + offsets.unsqueeze(1)).reshape(-1) + + out = torch.zeros_like(q) + if full_idx: + win_idx = _batched_windows(full_idx) + out.index_copy_( + 0, + win_idx, + F.scaled_dot_product_attention( + q.index_select(0, win_idx), + k.index_select(0, win_idx), + v.index_select(0, win_idx), + dropout_p=0.0, + is_causal=False, + ), + ) + if not groups: + return out + + out_nlhd = out.permute(0, 2, 1, 3) + for token_tuple, win_tuple in groups: + win_idx = _batched_windows(win_tuple) + token_idx = torch.tensor(token_tuple, device=q.device, dtype=torch.long) + attn = F.scaled_dot_product_attention( + q.index_select(0, win_idx).index_select(2, token_idx), + k.index_select(0, win_idx).index_select(2, token_idx), + v.index_select(0, win_idx).index_select(2, token_idx), + dropout_p=0.0, + is_causal=False, + ) + out_nlhd[win_idx[:, None], token_idx[None, :]] = attn.permute(0, 2, 1, 3) + return out + + +# --------------------------------------------------------------------------- +# Attention cores +# --------------------------------------------------------------------------- + + +class _QKVProj(nn.Module): + """Fused QKV projection.""" + + def __init__(self, dim: int, num_heads: int, qkv_bias: bool = True): + super().__init__() + assert dim % num_heads == 0 + self.dim = dim + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.qkv = nn.Linear(dim, 3 * dim, bias=qkv_bias) + + def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """x: (..., N, dim). Returns q, k, v each (..., num_heads, N, head_dim).""" + *B_dims, N, _ = x.shape + qkv = self.qkv(x).reshape(*B_dims, N, 3, self.num_heads, self.head_dim) + qkv = qkv.permute(*range(len(B_dims)), -3, -2, -4, -1).contiguous() + q, k, v = qkv.unbind(dim=-4) # each (..., num_heads, N, head_dim) + return q, k, v + + +class HeadwiseAttentionGate(nn.Module): + """Query-conditioned scalar gate per attention head. + + The gate is applied to SDPA output while heads are still separated: + (..., heads, N, head_dim). It uses a bias-free normal linear init, + matching the paper's + "extra q-projection logits" setup more closely than a hand-opened gate. + """ + + def __init__(self, dim: int, num_heads: int): + super().__init__() + self.dim = int(dim) + self.num_heads = int(num_heads) + self.proj = nn.Linear(self.dim, self.num_heads, bias=False) + + def forward(self, query_tokens: torch.Tensor, attn: torch.Tensor) -> torch.Tensor: + """query_tokens: (..., N, C), attn: (..., heads, N, head_dim).""" + gate = torch.sigmoid(self.proj(query_tokens)) + gate = gate.transpose(-1, -2).unsqueeze(-1).to(dtype=attn.dtype) + return attn * gate + + def grid(self, query_grid: torch.Tensor, dtype: torch.dtype) -> torch.Tensor: + """query_grid: (..., C). Returns (..., heads, 1).""" + gate = torch.sigmoid(self.proj(query_grid)).to(dtype=dtype) + return gate.unsqueeze(-1) + + +class QKVDepthwiseSpatial(nn.Module): + """Depthwise spatial filter on fused QKV maps in NHWC/NTHWC layout. + + The kernel is initialized to exact identity, so enabling this module is + nondisruptive for existing weights while still giving every off-center + tap an immediate gradient path. + """ + + def __init__(self, dim: int, kernel_size: int = 7): + super().__init__() + assert kernel_size > 0 and kernel_size % 2 == 1, ( + f"kernel_size must be a positive odd integer; got {kernel_size}") + self.dim = int(dim) + self.kernel_size = int(kernel_size) + self.pad = self.kernel_size // 2 + channels = 3 * self.dim + self.dw = nn.Conv2d(channels, channels, kernel_size=self.kernel_size, + padding=0, groups=channels, bias=True) + with torch.no_grad(): + self.dw.weight.zero_() + self.dw.weight[:, 0, self.pad, self.pad] = 1.0 + if self.dw.bias is not None: + self.dw.bias.zero_() + + def forward(self, qkv: torch.Tensor) -> torch.Tensor: + if qkv.dim() == 5: + B, T, H, W, C3 = qkv.shape + y = qkv.reshape(B * T, H, W, C3) + out_shape = (B, T, H, W, C3) + elif qkv.dim() == 4: + B, H, W, C3 = qkv.shape + y = qkv + out_shape = (B, H, W, C3) + else: + raise ValueError( + f"QKVDepthwiseSpatial expects NHWC or NTHWC, got {tuple(qkv.shape)}") + if C3 != 3 * self.dim: + raise ValueError(f"QKV channel count {C3} != 3*dim {3 * self.dim}") + y = y.permute(0, 3, 1, 2) + mode = 'reflect' if H > self.pad and W > self.pad else 'replicate' + + elems_per_item = C3 * (H + 2 * self.pad) * (W + 2 * self.pad) + max_items = max(1, 1_000_000_000 // max(1, elems_per_item)) + + def apply_dw(z: torch.Tensor) -> torch.Tensor: + z = F.pad(z, (self.pad, self.pad, self.pad, self.pad), mode=mode) + return self.dw(z) + + if y.shape[0] > max_items: + y = torch.cat([ + apply_dw(y[i:i + max_items]) + for i in range(0, y.shape[0], max_items) + ], dim=0) + else: + y = apply_dw(y) + y = y.permute(0, 2, 3, 1) + return y.reshape(out_shape) + + +class SwinAttention2D(nn.Module): + """2D Swin window attention with 2D RoPE on Q, K. + + Windows are non-overlapping; shift between consecutive blocks is the + caller's responsibility (via cyclic-shift before/after this module). + """ + + def __init__(self, dim: int, num_heads: int, window_size: int, + rope: Optional[RoPE2D] = None, + qkv_bias: bool = True, + qk_norm: bool = True, + qkv_dw_kernel: int = 0): + super().__init__() + self.dim = dim + self.num_heads = num_heads + self.window_size = window_size + self.head_dim = dim // num_heads + self.qkv = _QKVProj(dim, num_heads, qkv_bias=qkv_bias) + self.qkv_dw = ( + QKVDepthwiseSpatial(dim, kernel_size=int(qkv_dw_kernel)) + if qkv_dw_kernel and qkv_dw_kernel > 0 else None + ) + self.attn_gate = HeadwiseAttentionGate(dim, num_heads) + # QK-norm: RMSNorm applied to Q and K AFTER the qkv linear, BEFORE + # the dot product. Stabilizes attention at scale and in bf16 (where + # a large QK dot-product can overflow). Default on. + self.qk_norm = qk_norm + if qk_norm: + self.q_norm = RMSNorm(self.head_dim) + self.k_norm = RMSNorm(self.head_dim) + self.proj = nn.Linear(dim, dim) + # Zero-init output proj so residual branch contributes 0 at init. + # Paired with removal of LayerScale in SwinBlock2D. + nn.init.zeros_(self.proj.weight) + if self.proj.bias is not None: + nn.init.zeros_(self.proj.bias) + self.rope = rope or RoPE2D(self.head_dim) + + def forward( + self, + x: torch.Tensor, + attn_mask: torch.Tensor | None = None, + region_groups: tuple[int, tuple[int, ...], tuple[tuple[tuple[int, ...], tuple[int, ...]], ...]] | None = None, + region_ids: torch.Tensor | None = None, + ) -> torch.Tensor: + """x: (B, H, W, C). Returns (B, H, W, C). + + If (H, W) doesn't divide window_size, pads with zeros at bottom/right + up to the next multiple and masks padded keys out of attention. The + output is cropped back to (B, H, W, C). + """ + B, H, W, C = x.shape + w = self.window_size + pad_h = (-H) % w + pad_w = (-W) % w + has_pad = pad_h != 0 or pad_w != 0 + x_orig = x + if has_pad and self.qkv_dw is None: + # F.pad arg order for (B, H, W, C): (C_l, C_r, W_l, W_r, H_l, H_r) + x = F.pad(x, (0, 0, 0, pad_w, 0, pad_h)) + Hp, Wp = H + pad_h, W + pad_w + + if self.qkv_dw is not None: + N = w * w + nW_total = B * (Hp // w) * (Wp // w) + y = self.qkv.qkv(x_orig) + y = self.qkv_dw(y) + if has_pad: + y = F.pad(y, (0, 0, 0, pad_w, 0, pad_h)) + windows, _ = window_partition_2d(y, w) + qkv = (windows.reshape(nW_total, N, 3, self.num_heads, self.head_dim) + .permute(2, 0, 3, 1, 4) + .contiguous()) + q, k, v = qkv.unbind(dim=0) + gate_src = x_orig + if has_pad: + gate_src = F.pad(gate_src, (0, 0, 0, pad_w, 0, pad_h)) + gate_windows, _ = window_partition_2d(gate_src, w) + gate_tokens = gate_windows.reshape(nW_total, N, C) + else: + windows, _ = window_partition_2d(x, w) + nW_total = windows.shape[0] + N = w * w + tokens = windows.reshape(nW_total, N, C) + q, k, v = self.qkv(tokens) + gate_tokens = tokens + # q, k, v: (nW_total, num_heads, N, head_dim) + if self.qk_norm: + q = self.q_norm(q) + k = self.k_norm(k) + cos, sin = self.rope.get_cos_sin(w, w, device=x.device) + q = apply_rotary(q, cos, sin) + k = apply_rotary(k, cos, sin) + + if region_groups is not None: + attn = _sdpa_with_region_groups(q, k, v, *region_groups) + elif attn_mask is not None: + nW = int(attn_mask.shape[0]) + if q.shape[0] % nW != 0: + raise ValueError( + f"attention mask windows {nW} do not divide q batch {q.shape[0]}") + B_eff = q.shape[0] // nW + q_view = q.view(B_eff, nW, self.num_heads, N, self.head_dim) + k_view = k.view(B_eff, nW, self.num_heads, N, self.head_dim) + v_view = v.view(B_eff, nW, self.num_heads, N, self.head_dim) + attn = F.scaled_dot_product_attention( + q_view, k_view, v_view, + attn_mask=attn_mask.unsqueeze(0).unsqueeze(2), + dropout_p=0.0, is_causal=False) + attn = attn.reshape(nW_total, self.num_heads, N, self.head_dim) + elif region_ids is not None: + attn = _sdpa_with_region_ids(q, k, v, region_ids) + elif has_pad: + # Bool key-mask (True = real token). Broadcast over query dim L + # and heads via (B·nWin, 1, 1, N). Every window has ≥1 real token + # (pad_{h,w} < w), so softmax is always well-defined. + real = torch.zeros(Hp, Wp, dtype=torch.bool, device=x.device) + real[:H, :W] = True + nH_w, nW_w = Hp // w, Wp // w + real = real.view(nH_w, w, nW_w, w).permute(0, 2, 1, 3).contiguous() + per_win_mask = real.view(nH_w * nW_w, N) # (nWin, N) + attn_mask = per_win_mask.unsqueeze(0).expand(B, -1, -1).reshape(nW_total, 1, 1, N) + attn = F.scaled_dot_product_attention(q, k, v, + attn_mask=attn_mask, + dropout_p=0.0, is_causal=False) + else: + attn = F.scaled_dot_product_attention(q, k, v, dropout_p=0.0, is_causal=False) + + attn = self.attn_gate(gate_tokens, attn) + attn = attn.transpose(1, 2).reshape(nW_total, N, C) + attn = self.proj(attn) + + out_w = attn.reshape(nW_total, w, w, C) + out = window_unpartition_2d(out_w, w, (Hp, Wp), B) + if has_pad: + out = out[:, :H, :W, :].contiguous() + return out + + +class HaloAttention2D(nn.Module): + """Block-halo-overlapped 2D attention (HaloNet, Vaswani et al. 2021). + + Partitions input into non-overlapping `b×b` query blocks. Each block's + queries attend to a `(b+2h)×(b+2h)` halo'd region of keys/values — + the block itself plus `h` tokens of context on every side, reaching + into adjacent blocks (zero-padded at image boundaries). + + Output shape is still per-block `b×b`, so block-level output stride + is `b`, matching Swin semantics; but the attention context overlaps + between adjacent blocks by `2h` tokens per axis, so context boundaries + are soft rather than hard like in Swin. + + Combined with the caller's cyclic-shift alternation in the wrapping + block (same pattern as Swin), this gives "halo + shift" — within-block + halo context AND grid-averaging over a pair of blocks. See the model + design notes. + + h=0 reduces to plain Swin windowed attention (without shift). Use + from SwinBlock2D with `attn_class=HaloAttention2D` and non-zero halo. + + Masking: + • Block-alignment padding (right/bottom of the grid) → masked out. + • Halo tokens that fall outside the original H×W image → masked out + (they would otherwise contribute the k-projection of zero-padded + values, i.e. just the k-linear bias, to real queries' attention). + """ + + def __init__(self, dim: int, num_heads: int, block_size: int, halo: int, + rope: Optional[RoPE2D] = None, + qkv_bias: bool = True, qk_norm: bool = True): + super().__init__() + assert halo >= 0 and block_size > 0, f"block_size={block_size} halo={halo}" + self.dim = dim + self.num_heads = num_heads + self.block_size = block_size + self.halo = halo + self.head_dim = dim // num_heads + self.qkv = _QKVProj(dim, num_heads, qkv_bias=qkv_bias) + self.attn_gate = HeadwiseAttentionGate(dim, num_heads) + self.qk_norm = qk_norm + if qk_norm: + self.q_norm = RMSNorm(self.head_dim) + self.k_norm = RMSNorm(self.head_dim) + self.proj = nn.Linear(dim, dim) + nn.init.zeros_(self.proj.weight) + if self.proj.bias is not None: + nn.init.zeros_(self.proj.bias) + # RoPE lives in the (b+2h)×(b+2h) halo'd coord system. Q uses the + # central b×b subset of this coord system; K/V use the full set. + self.rope = rope or RoPE2D(self.head_dim) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """x: (B, H, W, C). Returns (B, H, W, C).""" + B, H, W, C = x.shape + b = self.block_size + h = self.halo + kh = b + 2 * h + + # 1. Pad to block alignment on bottom/right. + pad_h = (-H) % b + pad_w = (-W) % b + has_block_pad = pad_h != 0 or pad_w != 0 + if has_block_pad: + x_aligned = F.pad(x, (0, 0, 0, pad_w, 0, pad_h)) + else: + x_aligned = x + Hp, Wp = H + pad_h, W + pad_w + nH, nW = Hp // b, Wp // b + nBlocks = nH * nW + + # 2. QKV projection on the block-aligned (no-halo) input. + # Output split into separate q_full, k_full, v_full because the + # block extraction differs between Q and K/V. + heads = self.num_heads + D = self.head_dim + qkv = self.qkv.qkv(x_aligned) # (B, Hp, Wp, 3·heads·D) + qkv = qkv.view(B, Hp, Wp, 3, heads, D) + q_full = qkv[..., 0, :, :] # (B, Hp, Wp, heads, D) + k_full = qkv[..., 1, :, :] + v_full = qkv[..., 2, :, :] + + # 3. Q: block partition (no halo). Output (B·nBlocks, heads, b², D). + Q = q_full.view(B, nH, b, nW, b, heads, D) + Q = Q.permute(0, 1, 3, 5, 2, 4, 6).contiguous() # (B, nH, nW, heads, b, b, D) + Q = Q.view(B * nBlocks, heads, b * b, D) + q_tokens = (x_aligned.view(B, nH, b, nW, b, C) + .permute(0, 1, 3, 2, 4, 5) + .reshape(B * nBlocks, b * b, C)) + + # 4. K, V: halo-pad by `h` on spatial dims, then unfold (b+2h)×(b+2h) + # windows with stride b. Zero-pad at boundaries. + if h > 0: + k_pad = F.pad(k_full, (0, 0, 0, 0, h, h, h, h)) + v_pad = F.pad(v_full, (0, 0, 0, 0, h, h, h, h)) + else: + k_pad = k_full + v_pad = v_full + # k_pad: (B, Hp+2h, Wp+2h, heads, D) + # unfold(1, kh, b) → (B, nH, Wp+2h, heads, D, kh) + # unfold(2, kh, b) → (B, nH, nW, heads, D, kh, kh) + # permute to (B, nH, nW, heads, kh, kh, D) + K = k_pad.unfold(1, kh, b).unfold(2, kh, b).permute(0, 1, 2, 3, 5, 6, 4).contiguous() + V = v_pad.unfold(1, kh, b).unfold(2, kh, b).permute(0, 1, 2, 3, 5, 6, 4).contiguous() + N_kv = kh * kh + K = K.view(B * nBlocks, heads, N_kv, D) + V = V.view(B * nBlocks, heads, N_kv, D) + + # 5. QK-norm + RoPE. K/V use full (kh × kh) coord tables; Q uses the + # central b×b subset of those tables. + if self.qk_norm: + Q = self.q_norm(Q) + K = self.k_norm(K) + cos_kv, sin_kv = self.rope.get_cos_sin(kh, kh, device=x.device) + # Extract central b×b rows of the flattened (kh·kh, D//2) table. + device = cos_kv.device + rows = torch.arange(h, h + b, device=device) + cols = torch.arange(h, h + b, device=device) + central_idx = (rows.unsqueeze(1) * kh + cols.unsqueeze(0)).flatten() + cos_q = cos_kv.index_select(0, central_idx) + sin_q = sin_kv.index_select(0, central_idx) + Q = apply_rotary(Q, cos_q, sin_q) + K = apply_rotary(K, cos_kv, sin_kv) + + # 6. Attention mask for padding + out-of-image halo. + # Build an (Hp+2h, Wp+2h) "valid" map (True = inside original + # H×W image), unfold into per-block (kh, kh) key masks. + needs_mask = has_block_pad or h > 0 + if needs_mask: + valid = torch.zeros(Hp + 2 * h, Wp + 2 * h, dtype=torch.bool, device=x.device) + valid[h:h + H, h:h + W] = True + valid_u = valid.unfold(0, kh, b).unfold(1, kh, b) # (nH, nW, kh, kh) + valid_u = valid_u.reshape(nBlocks, N_kv) + attn_mask = valid_u.unsqueeze(0).expand(B, -1, -1).reshape(B * nBlocks, 1, 1, N_kv) + attn = F.scaled_dot_product_attention(Q, K, V, attn_mask=attn_mask, + dropout_p=0.0, is_causal=False) + else: + attn = F.scaled_dot_product_attention(Q, K, V, dropout_p=0.0, is_causal=False) + + # 7. Unpack: (B·nBlocks, heads, b², D) → (B, Hp, Wp, C) → crop to (B, H, W, C) + attn = self.attn_gate(q_tokens, attn) + attn = attn.transpose(1, 2).reshape(B * nBlocks, b * b, heads * D) # (BnBlocks, b², C) + attn = self.proj(attn) + out = attn.view(B, nH, nW, b, b, heads * D) + out = out.permute(0, 1, 3, 2, 4, 5).contiguous() + out = out.view(B, Hp, Wp, heads * D) + if has_block_pad: + out = out[:, :H, :W, :].contiguous() + return out + + +class SwinAttention3D(nn.Module): + """3D Swin window attention with 3D RoPE on Q, K. + + Spatial windows are non-overlapping (H_w × W_w); the temporal axis is + ALWAYS used in full (T_w = T). This means every window has shape + (T, H_w, W_w) and attention is fully-temporal + windowed-spatial. + + Rationale: at our working resolutions (T in 3–48, spatial grid at /16), + windowed temporal would either leave temporal context uncovered + (single-frame windows) or add state-management complexity (temporal + shifts interacting with variable T). Full-T within-window is cheap — + attention scales as H_w²·W_w²·T², and at T≤48 with H_w=W_w=8 that's + under 10M per window per head even at the worst T. + """ + + def __init__(self, dim: int, num_heads: int, + window_H: int, window_W: int, + rope: Optional[RoPE3D] = None, + qkv_bias: bool = True, + qk_norm: bool = True, + zero_init_proj: bool = True, + qkv_dw_kernel: int = 0): + super().__init__() + self.dim = dim + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.window_H = window_H + self.window_W = window_W + self.qkv = _QKVProj(dim, num_heads, qkv_bias=qkv_bias) + self.qkv_dw = ( + QKVDepthwiseSpatial(dim, kernel_size=int(qkv_dw_kernel)) + if qkv_dw_kernel and qkv_dw_kernel > 0 else None + ) + self.attn_gate = HeadwiseAttentionGate(dim, num_heads) + self.qk_norm = qk_norm + if qk_norm: + self.q_norm = RMSNorm(self.head_dim) + self.k_norm = RMSNorm(self.head_dim) + self.proj = nn.Linear(dim, dim) + if zero_init_proj: + # Zero-init output proj so residual branch contributes 0 at init. + nn.init.zeros_(self.proj.weight) + if self.proj.bias is not None: + nn.init.zeros_(self.proj.bias) + self.rope = rope or RoPE3D(self.head_dim) + + def forward( + self, + x: torch.Tensor, + t_stride: float = 1.0, + region_groups: tuple[int, tuple[int, ...], tuple[tuple[tuple[int, ...], tuple[int, ...]], ...]] | None = None, + region_ids: torch.Tensor | None = None, + ) -> torch.Tensor: + """x: (B, T, H, W, C). Returns (B, T, H, W, C). + + If (H, W) doesn't divide the spatial window, pads with zeros at + bottom/right and masks padded keys out of attention. T is always + used in full (T_w = T, no temporal padding). + """ + B, T, H, W, C = x.shape + T_w = T + H_w = self.window_H + W_w = self.window_W + pad_h = (-H) % H_w + pad_w = (-W) % W_w + has_pad = pad_h != 0 or pad_w != 0 + x_orig = x + if has_pad and self.qkv_dw is None: + # F.pad order for (B, T, H, W, C): (C_l, C_r, W_l, W_r, H_l, H_r) + x = F.pad(x, (0, 0, 0, pad_w, 0, pad_h)) + Hp, Wp = H + pad_h, W + pad_w + + N = T_w * H_w * W_w + nH_w, nW_w = Hp // H_w, Wp // W_w + nW_total = B * nH_w * nW_w + window_batch = int(getattr(self, 'inference_window_batch', 0) or 0) + if window_batch <= 0 and not self.training: + elems_per_window = N * C + max_windows = max(1, 200_000_000 // max(1, elems_per_window)) + if nW_total > max_windows: + window_batch = max_windows + + def _slice_region_groups(start: int, end: int): + if region_groups is None: + return None + if B != 1: + return region_groups + num_windows, full_idx, groups = region_groups + if num_windows != nW_total: + return region_groups + full = tuple(idx - start for idx in full_idx + if start <= idx < end) + sliced = [] + for token_tuple, win_tuple in groups: + wins = tuple(idx - start for idx in win_tuple + if start <= idx < end) + if wins: + sliced.append((token_tuple, wins)) + return (end - start, full, tuple(sliced)) + + pad_attn_mask = None + if has_pad and region_groups is None and region_ids is None: + # Spatial validity (per window) tiled across T (T never padded). + real = torch.zeros(Hp, Wp, dtype=torch.bool, device=x.device) + real[:H, :W] = True + real = real.view(nH_w, H_w, nW_w, W_w).permute(0, 2, 1, 3).contiguous() + per_win_hw = real.view(nH_w * nW_w, H_w * W_w) + per_win_mask = per_win_hw.unsqueeze(1).expand(-1, T_w, -1) + per_win_mask = per_win_mask.reshape(nH_w * nW_w, N) + pad_attn_mask = per_win_mask.unsqueeze(0).expand( + B, -1, -1).reshape(nW_total, 1, 1, N) + + # Local positions per window: t=0..T-1 (full), h=0..H_w-1, w=0..W_w-1. + # Temporal inter-window info isn't needed since T_w=T (no temporal + # partitioning). Spatial inter-window info is handled by the + # SwinBlock3D wrapper's cyclic shift. + cos, sin = self.rope.get_cos_sin(T_w, H_w, W_w, device=x.device, + t_stride=t_stride) + + def _take_windows(grid: torch.Tensor, start: int, end: int) -> torch.Tensor: + # grid: (B, T, Hp, Wp, Cg). Return only requested spatial windows. + Cg = grid.shape[-1] + view = grid.view(B, T_w, nH_w, H_w, nW_w, W_w, Cg) + out = [] + for idx in range(start, end): + b = idx // (nH_w * nW_w) + sp = idx - b * (nH_w * nW_w) + ih = sp // nW_w + iw = sp - ih * nW_w + out.append(view[b, :, ih, :, iw, :, :]) + return torch.stack(out, dim=0).contiguous() + + def _attend_windows(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, + gate_tokens: torch.Tensor, + start: int, end: int) -> torch.Tensor: + if self.qk_norm: + q = self.q_norm(q) + k = self.k_norm(k) + q = apply_rotary(q, cos, sin) + k = apply_rotary(k, cos, sin) + if region_groups is not None: + groups = _slice_region_groups(start, end) + attn = _sdpa_with_region_groups(q, k, v, *groups) + elif region_ids is not None: + attn = _sdpa_with_region_ids(q, k, v, region_ids[start:end]) + elif pad_attn_mask is not None: + attn = F.scaled_dot_product_attention( + q, k, v, attn_mask=pad_attn_mask[start:end], + dropout_p=0.0, is_causal=False) + else: + attn = F.scaled_dot_product_attention( + q, k, v, dropout_p=0.0, is_causal=False) + attn = self.attn_gate(gate_tokens, attn) + attn = attn.transpose(1, 2).reshape(end - start, N, C) + return self.proj(attn) + + use_lazy_window_path = ( + window_batch > 0 and not self.training and nW_total > window_batch + and (region_groups is None or B == 1) + ) + if use_lazy_window_path: + gate_src = x_orig + if has_pad: + gate_src = F.pad(gate_src, (0, 0, 0, pad_w, 0, pad_h)) + qkv_grid = None + if self.qkv_dw is not None: + frame_chunk = int( + getattr(self, 'inference_qkv_frame_chunk', 0) or 0) + frame_chunk = max(1, frame_chunk) if frame_chunk > 0 else T_w + for t_start in range(0, T_w, frame_chunk): + t_end = min(t_start + frame_chunk, T_w) + y = self.qkv.qkv(x_orig[:, t_start:t_end]) + y = self.qkv_dw(y) + if has_pad: + y = F.pad(y, (0, 0, 0, pad_w, 0, pad_h)) + if qkv_grid is None: + qkv_grid = y.new_empty(B, T_w, Hp, Wp, 3 * C) + qkv_grid[:, t_start:t_end] = y + else: + qkv_grid = None + + attn = gate_src.new_empty((nW_total, N, C)) + for start in range(0, nW_total, window_batch): + end = min(start + window_batch, nW_total) + gate_tokens = _take_windows( + gate_src, start, end).reshape(end - start, N, C) + if self.qkv_dw is not None: + qkv_win = _take_windows(qkv_grid, start, end) + qkv = (qkv_win.reshape( + end - start, N, 3, self.num_heads, + self.head_dim) + .permute(2, 0, 3, 1, 4) + .contiguous()) + q, k, v = qkv.unbind(dim=0) + else: + q, k, v = self.qkv(gate_tokens) + attn[start:end] = _attend_windows( + q, k, v, gate_tokens, start, end) + + out_w = attn.reshape(nW_total, T_w, H_w, W_w, C) + out = window_unpartition_3d(out_w, T_w, H_w, W_w, (T, Hp, Wp), B) + if has_pad: + out = out[:, :, :H, :W, :].contiguous() + return out + + if self.qkv_dw is not None: + y = self.qkv.qkv(x_orig) + y = self.qkv_dw(y) + if has_pad: + y = F.pad(y, (0, 0, 0, pad_w, 0, pad_h)) + windows, _ = window_partition_3d(y, T_w, H_w, W_w) + qkv_tokens = windows.reshape( + nW_total, N, 3, self.num_heads, self.head_dim) + gate_src = x_orig + if has_pad: + gate_src = F.pad(gate_src, (0, 0, 0, pad_w, 0, pad_h)) + gate_windows, _ = window_partition_3d(gate_src, T_w, H_w, W_w) + gate_tokens = gate_windows.reshape(nW_total, N, C) + else: + windows, _ = window_partition_3d(x, T_w, H_w, W_w) + nW_total = windows.shape[0] + tokens = windows.reshape(nW_total, N, C) + gate_tokens = tokens + + def _run_window_slice(start: int, end: int) -> torch.Tensor: + if self.qkv_dw is not None: + qkv = qkv_tokens[start:end].permute( + 2, 0, 3, 1, 4).contiguous() + q, k, v = qkv.unbind(dim=0) + else: + q, k, v = self.qkv(tokens[start:end]) + if self.qk_norm: + q = self.q_norm(q) + k = self.k_norm(k) + q = apply_rotary(q, cos, sin) + k = apply_rotary(k, cos, sin) + if region_groups is not None: + groups = _slice_region_groups(start, end) + attn = _sdpa_with_region_groups(q, k, v, *groups) + elif region_ids is not None: + attn = _sdpa_with_region_ids(q, k, v, region_ids[start:end]) + elif pad_attn_mask is not None: + attn = F.scaled_dot_product_attention( + q, k, v, attn_mask=pad_attn_mask[start:end], + dropout_p=0.0, is_causal=False) + else: + attn = F.scaled_dot_product_attention( + q, k, v, dropout_p=0.0, is_causal=False) + attn = self.attn_gate(gate_tokens[start:end], attn) + attn = attn.transpose(1, 2).reshape(end - start, N, C) + return self.proj(attn) + + can_batch_windows = ( + window_batch > 0 and not self.training and nW_total > window_batch + and (region_groups is None or B == 1) + ) + if can_batch_windows: + attn = gate_tokens.new_empty((nW_total, N, C)) + for start in range(0, nW_total, window_batch): + end = min(start + window_batch, nW_total) + attn[start:end] = _run_window_slice(start, end) + else: + attn = _run_window_slice(0, nW_total) + out_w = attn.reshape(nW_total, T_w, H_w, W_w, C) + out = window_unpartition_3d(out_w, T_w, H_w, W_w, (T, Hp, Wp), B) + if has_pad: + out = out[:, :, :H, :W, :].contiguous() + return out + + +class HaloAttention3D(nn.Module): + """Block-halo-overlapped 3D attention (spatial-halo, full-temporal). + + Spatial extension of HaloAttention2D: H, W are block-partitioned at + stride `block_H`, `block_W` with halo `h` on each side; T is always + used in full (matches Swin-3D's full-T convention). Each block's + queries are `b_H · b_W · T` tokens attending to + `(b_H+2h) · (b_W+2h) · T` tokens. + + h=0 reduces to plain Swin-3D windowed attention (without shift). + """ + + def __init__(self, dim: int, num_heads: int, + block_H: int, block_W: int, halo: int, + rope: Optional[RoPE3D] = None, + qkv_bias: bool = True, qk_norm: bool = True): + super().__init__() + assert halo >= 0, f"halo={halo} must be ≥ 0" + self.dim = dim + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.block_H = block_H + self.block_W = block_W + self.halo = halo + self.qkv = _QKVProj(dim, num_heads, qkv_bias=qkv_bias) + self.attn_gate = HeadwiseAttentionGate(dim, num_heads) + self.qk_norm = qk_norm + if qk_norm: + self.q_norm = RMSNorm(self.head_dim) + self.k_norm = RMSNorm(self.head_dim) + self.proj = nn.Linear(dim, dim) + nn.init.zeros_(self.proj.weight) + if self.proj.bias is not None: + nn.init.zeros_(self.proj.bias) + self.rope = rope or RoPE3D(self.head_dim) + + def forward(self, x: torch.Tensor, t_stride: float = 1.0) -> torch.Tensor: + """x: (B, T, H, W, C). Returns (B, T, H, W, C).""" + B, T, H, W, C = x.shape + bH, bW = self.block_H, self.block_W + h = self.halo + khH, khW = bH + 2 * h, bW + 2 * h + heads, D = self.num_heads, self.head_dim + + # 1. Block-align pad on H, W (T stays full). + pad_h = (-H) % bH + pad_w = (-W) % bW + has_block_pad = pad_h != 0 or pad_w != 0 + if has_block_pad: + x_a = F.pad(x, (0, 0, 0, pad_w, 0, pad_h)) + else: + x_a = x + Hp, Wp = H + pad_h, W + pad_w + nH, nW = Hp // bH, Wp // bW + nBlocks = nH * nW + + # 2. QKV projection. + qkv = self.qkv.qkv(x_a) # (B, T, Hp, Wp, 3·heads·D) + qkv = qkv.view(B, T, Hp, Wp, 3, heads, D) + q_full = qkv[..., 0, :, :] # (B, T, Hp, Wp, heads, D) + k_full = qkv[..., 1, :, :] + v_full = qkv[..., 2, :, :] + + # 3. Q: block partition (no halo). Per block: T·bH·bW tokens. + # (B, T, Hp, Wp, heads, D) → (B, T, nH, bH, nW, bW, heads, D) + # → permute to (B, nH, nW, heads, T, bH, bW, D) + Q = q_full.view(B, T, nH, bH, nW, bW, heads, D) + Q = Q.permute(0, 2, 4, 6, 1, 3, 5, 7).contiguous() + Q = Q.view(B * nBlocks, heads, T * bH * bW, D) + q_tokens = (x_a.view(B, T, nH, bH, nW, bW, C) + .permute(0, 2, 4, 1, 3, 5, 6) + .reshape(B * nBlocks, T * bH * bW, C)) + + # 4. K, V: halo pad spatially (T always full), then unfold. + if h > 0: + k_pad = F.pad(k_full, (0, 0, 0, 0, h, h, h, h)) + v_pad = F.pad(v_full, (0, 0, 0, 0, h, h, h, h)) + else: + k_pad = k_full + v_pad = v_full + # Shape: (B, T, Hp+2h, Wp+2h, heads, D) + # unfold(2, khH, bH) → (B, T, nH, Wp+2h, heads, D, khH) + # unfold(3, khW, bW) → (B, T, nH, nW, heads, D, khH, khW) + # permute to (B, nH, nW, heads, T, khH, khW, D) + K = k_pad.unfold(2, khH, bH).unfold(3, khW, bW) + V = v_pad.unfold(2, khH, bH).unfold(3, khW, bW) + K = K.permute(0, 2, 3, 4, 1, 6, 7, 5).contiguous() + V = V.permute(0, 2, 3, 4, 1, 6, 7, 5).contiguous() + N_kv = T * khH * khW + K = K.view(B * nBlocks, heads, N_kv, D) + V = V.view(B * nBlocks, heads, N_kv, D) + + # 5. QK-norm + RoPE. K/V use full (T, khH, khW) coord tables. + # Q uses (T full, H in [h, h+bH), W in [h, h+bW)) subset. + if self.qk_norm: + Q = self.q_norm(Q) + K = self.k_norm(K) + cos_kv, sin_kv = self.rope.get_cos_sin(T, khH, khW, + device=x.device, + t_stride=t_stride) + dev = cos_kv.device + t_idx = torch.arange(T, device=dev) + r_idx = torch.arange(h, h + bH, device=dev) + c_idx = torch.arange(h, h + bW, device=dev) + # Flat row-major index into (T, khH, khW) grid: + flat_q_idx = (t_idx[:, None, None] * (khH * khW) + + r_idx[None, :, None] * khW + + c_idx[None, None, :]).reshape(-1) + cos_q = cos_kv.index_select(0, flat_q_idx) + sin_q = sin_kv.index_select(0, flat_q_idx) + Q = apply_rotary(Q, cos_q, sin_q) + K = apply_rotary(K, cos_kv, sin_kv) + + # 6. Attention mask for block-pad + out-of-image halo. + needs_mask = has_block_pad or h > 0 + if needs_mask: + valid_hw = torch.zeros(Hp + 2 * h, Wp + 2 * h, + dtype=torch.bool, device=x.device) + valid_hw[h:h + H, h:h + W] = True + valid_u = valid_hw.unfold(0, khH, bH).unfold(1, khW, bW) # (nH, nW, khH, khW) + valid_u = valid_u.reshape(nBlocks, khH * khW) + # Tile over T (T always valid). + valid_tiled = valid_u.unsqueeze(1).expand(-1, T, -1).reshape(nBlocks, N_kv) + attn_mask = valid_tiled.unsqueeze(0).expand(B, -1, -1).reshape(B * nBlocks, 1, 1, N_kv) + attn = F.scaled_dot_product_attention(Q, K, V, attn_mask=attn_mask, + dropout_p=0.0, is_causal=False) + else: + attn = F.scaled_dot_product_attention(Q, K, V, + dropout_p=0.0, is_causal=False) + + # 7. Unpack: (B·nBlocks, heads, T·bH·bW, D) → (B, T, H, W, C) + attn = self.attn_gate(q_tokens, attn) + attn = attn.transpose(1, 2).reshape(B * nBlocks, T * bH * bW, heads * D) + attn = self.proj(attn) + out = attn.view(B, nH, nW, T, bH, bW, heads * D) + out = out.permute(0, 3, 1, 4, 2, 5, 6).contiguous() + out = out.view(B, T, Hp, Wp, heads * D) + if has_block_pad: + out = out[:, :, :H, :W, :].contiguous() + return out + + +class AxialAttention2D(nn.Module): + """Axial (pure row or pure column) 2D attention. + + For `axis='H'` (VERTICAL): each token attends to all tokens in its own + column (same W, all H). Input reshape: (B, H, W, C) → (B·W, H, C). + For `axis='W'` (HORIZONTAL): row-wise. (B, H, W, C) → (B·H, W, C). + + Uses shared RoPE2D with a trivial (size-1) axis on the un-attended + direction — half the head_dim contributes positional modulation along + the attended axis, half is pure dot-product. Augmentations (rescale, + jitter) from `refresh_augs()` apply uniformly alongside the dense + Swin blocks using the same RoPE instance. + """ + + def __init__(self, dim: int, num_heads: int, axis: str, + rope: Optional[RoPE2D] = None, + qkv_bias: bool = True, qk_norm: bool = True): + super().__init__() + assert axis in ('H', 'W'), f"axis must be 'H' or 'W', got {axis!r}" + self.dim = dim + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.axis = axis + self.qkv = _QKVProj(dim, num_heads, qkv_bias=qkv_bias) + self.attn_gate = HeadwiseAttentionGate(dim, num_heads) + self.qk_norm = qk_norm + if qk_norm: + self.q_norm = RMSNorm(self.head_dim) + self.k_norm = RMSNorm(self.head_dim) + self.proj = nn.Linear(dim, dim) + nn.init.zeros_(self.proj.weight) + if self.proj.bias is not None: + nn.init.zeros_(self.proj.bias) + self.rope = rope or RoPE2D(self.head_dim) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """x: (B, H, W, C). Returns (B, H, W, C).""" + B, H, W, C = x.shape + # Reshape so the attended axis is the sequence dim; the other spatial + # axis joins the batch. Nothing else needs padding — any H or W works. + if self.axis == 'H': + # Columns: each of W columns has H tokens. + # (B, H, W, C) → (B, W, H, C) → (B·W, H, C) + x_strips = x.permute(0, 2, 1, 3).reshape(B * W, H, C) + N = H + num_h_rope, num_w_rope = H, 1 + else: # 'W' + # Rows: each of H rows has W tokens. + # (B, H, W, C) → (B·H, W, C) + x_strips = x.reshape(B * H, W, C) + N = W + num_h_rope, num_w_rope = 1, W + + q, k, v = self.qkv(x_strips) # (B·M, heads, N, head_dim) + if self.qk_norm: + q = self.q_norm(q) + k = self.k_norm(k) + cos, sin = self.rope.get_cos_sin(num_h_rope, num_w_rope, device=x.device) + q = apply_rotary(q, cos, sin) + k = apply_rotary(k, cos, sin) + + attn = F.scaled_dot_product_attention(q, k, v, dropout_p=0.0, is_causal=False) + attn = self.attn_gate(x_strips, attn) + # (B·M, heads, N, head_dim) → (B·M, N, C) + attn = attn.transpose(1, 2).reshape(-1, N, C) + attn = self.proj(attn) + + # Un-reshape back to (B, H, W, C) + if self.axis == 'H': + out = attn.view(B, W, H, C).permute(0, 2, 1, 3).contiguous() + else: + out = attn.view(B, H, W, C) + return out + + +class AxialAttention3D(nn.Module): + """Axial 3D attention with full T. + + For `axis='H'` (VERTICAL): each token attends to all (t', h', w_fixed) + with t' ∈ [0,T), h' ∈ [0,H) — i.e., T·H tokens per column-strip. + For `axis='W'` (HORIZONTAL): per row-strip of T·W tokens. + + Uses shared RoPE3D with num=1 on the un-attended spatial axis. T is + always full, scaled by `t_stride` just like Swin-3D and FullST. + """ + + def __init__(self, dim: int, num_heads: int, axis: str, + rope: Optional[RoPE3D] = None, + qkv_bias: bool = True, qk_norm: bool = True): + super().__init__() + assert axis in ('H', 'W'), f"axis must be 'H' or 'W', got {axis!r}" + self.dim = dim + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.axis = axis + self.qkv = _QKVProj(dim, num_heads, qkv_bias=qkv_bias) + self.attn_gate = HeadwiseAttentionGate(dim, num_heads) + self.qk_norm = qk_norm + if qk_norm: + self.q_norm = RMSNorm(self.head_dim) + self.k_norm = RMSNorm(self.head_dim) + self.proj = nn.Linear(dim, dim) + nn.init.zeros_(self.proj.weight) + if self.proj.bias is not None: + nn.init.zeros_(self.proj.bias) + self.rope = rope or RoPE3D(self.head_dim) + + def forward(self, x: torch.Tensor, t_stride: float = 1.0) -> torch.Tensor: + """x: (B, T, H, W, C). Returns (B, T, H, W, C).""" + B, T, H, W, C = x.shape + if self.axis == 'H': + # Strip per W column: (T·H) tokens. Reshape: + # (B, T, H, W, C) → (B, W, T, H, C) → (B·W, T·H, C) + x_strips = x.permute(0, 3, 1, 2, 4).reshape(B * W, T * H, C) + N = T * H + num_t_rope, num_h_rope, num_w_rope = T, H, 1 + else: # 'W' + # (B, T, H, W, C) → (B, H, T, W, C) → (B·H, T·W, C) + x_strips = x.permute(0, 2, 1, 3, 4).reshape(B * H, T * W, C) + N = T * W + num_t_rope, num_h_rope, num_w_rope = T, 1, W + + q, k, v = self.qkv(x_strips) + if self.qk_norm: + q = self.q_norm(q) + k = self.k_norm(k) + cos, sin = self.rope.get_cos_sin(num_t_rope, num_h_rope, num_w_rope, + device=x.device, t_stride=t_stride) + q = apply_rotary(q, cos, sin) + k = apply_rotary(k, cos, sin) + + attn = F.scaled_dot_product_attention(q, k, v, dropout_p=0.0, is_causal=False) + attn = self.attn_gate(x_strips, attn) + attn = attn.transpose(1, 2).reshape(-1, N, C) + attn = self.proj(attn) + + if self.axis == 'H': + out = attn.view(B, W, T, H, C).permute(0, 2, 3, 1, 4).contiguous() + else: + out = attn.view(B, H, T, W, C).permute(0, 2, 1, 3, 4).contiguous() + return out + + +class CSwinAttention2D(nn.Module): + """Cross-shaped window attention (CSwin, Dong et al. 2022) — 2D. + + Heads split evenly: the first half attend along H (vertical stripes — + each token sees its full column), the second half attend along W + (horizontal stripes — each token sees its full row). Within one + forward pass every output token has been mixed along BOTH axes + (different heads carry the different mixings, then the output proj + blends them together). + + State-dict layout intentionally identical to AxialAttention2D: + qkv.qkv.{weight,bias}, q_norm.weight, k_norm.weight, proj.{weight,bias} + This lets a weight-permutation surgery convert an axial checkpoint + into a CSwin one without remapping keys. + """ + + def __init__(self, dim: int, num_heads: int, + rope: Optional[RoPE2D] = None, + qkv_bias: bool = True, qk_norm: bool = True, + qkv_dw_kernel: int = 0): + super().__init__() + assert num_heads % 2 == 0, ( + f"CSwinAttention2D requires num_heads % 2 == 0, got {num_heads}") + self.dim = dim + self.num_heads = num_heads + self.h_heads = num_heads // 2 # V-stripe heads (attend along H) + self.w_heads = num_heads - self.h_heads # H-stripe heads (attend along W) + self.head_dim = dim // num_heads + self.qkv = _QKVProj(dim, num_heads, qkv_bias=qkv_bias) + self.qkv_dw = ( + QKVDepthwiseSpatial(dim, kernel_size=int(qkv_dw_kernel)) + if qkv_dw_kernel and qkv_dw_kernel > 0 else None + ) + self.attn_gate = HeadwiseAttentionGate(dim, num_heads) + self.qk_norm = qk_norm + if qk_norm: + self.q_norm = RMSNorm(self.head_dim) + self.k_norm = RMSNorm(self.head_dim) + self.proj = nn.Linear(dim, dim) + nn.init.zeros_(self.proj.weight) + if self.proj.bias is not None: + nn.init.zeros_(self.proj.bias) + self.rope = rope or RoPE2D(self.head_dim) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """x: (B, H, W, C). Returns (B, H, W, C).""" + B, H, W, C = x.shape + h_heads, w_heads, D = self.h_heads, self.w_heads, self.head_dim + + # QKV is pointwise unless qkv_dw is enabled; the optional DW filter + # runs on the full spatial map before strip reshape. + if self.qkv_dw is not None: + y = self.qkv_dw(self.qkv.qkv(x)) + qkv = (y.reshape(B, H * W, 3, self.num_heads, D) + .permute(2, 0, 3, 1, 4) + .contiguous()) + q, k, v = qkv.unbind(dim=0) + else: + x_flat = x.reshape(B, H * W, C) + q, k, v = self.qkv(x_flat) # each (B, num_heads, H*W, D) + if self.qk_norm: + q = self.q_norm(q) + k = self.k_norm(k) + + # Split heads into V-stripe (attend H) and H-stripe (attend W) groups. + q_h, k_h, v_h = q[:, :h_heads], k[:, :h_heads], v[:, :h_heads] + q_w, k_w, v_w = q[:, h_heads:], k[:, h_heads:], v[:, h_heads:] + + # H-group: each of W columns is a strip of H tokens. + # (B, h_heads, H*W, D) → (B, h_heads, H, W, D) → (B, W, h_heads, H, D) → (B·W, h_heads, H, D) + def _to_hstrip(t): + return (t.reshape(B, h_heads, H, W, D) + .permute(0, 3, 1, 2, 4) + .reshape(B * W, h_heads, H, D)) + q_h, k_h, v_h = _to_hstrip(q_h), _to_hstrip(k_h), _to_hstrip(v_h) + + # W-group: each of H rows is a strip of W tokens. + def _to_wstrip(t): + return (t.reshape(B, w_heads, H, W, D) + .permute(0, 2, 1, 3, 4) + .reshape(B * H, w_heads, W, D)) + q_w, k_w, v_w = _to_wstrip(q_w), _to_wstrip(k_w), _to_wstrip(v_w) + + # RoPE: H-group uses the H-axis coords (num_w=1 → identity on W-chunk); + # W-group uses the W-axis coords (num_h=1 → identity on H-chunk). + cos_h, sin_h = self.rope.get_cos_sin(H, 1, device=x.device) + cos_w, sin_w = self.rope.get_cos_sin(1, W, device=x.device) + q_h = apply_rotary(q_h, cos_h, sin_h) + k_h = apply_rotary(k_h, cos_h, sin_h) + q_w = apply_rotary(q_w, cos_w, sin_w) + k_w = apply_rotary(k_w, cos_w, sin_w) + + # Attention along each axis. + attn_h = F.scaled_dot_product_attention(q_h, k_h, v_h, + dropout_p=0.0, is_causal=False) + attn_w = F.scaled_dot_product_attention(q_w, k_w, v_w, + dropout_p=0.0, is_causal=False) + + # Un-strip back to (B, H, W, head_group, D) and concat heads. + # H-group: (B·W, h_heads, H, D) → (B, W, h_heads, H, D) → (B, H, W, h_heads, D) + attn_h = (attn_h.reshape(B, W, h_heads, H, D) + .permute(0, 3, 1, 2, 4) + .contiguous()) + # W-group: (B·H, w_heads, W, D) → (B, H, w_heads, W, D) → (B, H, W, w_heads, D) + attn_w = (attn_w.reshape(B, H, w_heads, W, D) + .permute(0, 1, 3, 2, 4) + .contiguous()) + + attn = torch.cat([attn_h, attn_w], dim=-2) + attn = attn * self.attn_gate.grid(x, dtype=attn.dtype) + attn = attn.reshape(B, H, W, C) + return self.proj(attn) + + +class CSwinAttention3D(nn.Module): + """Cross-shaped window attention — 3D (full temporal + split spatial). + + Heads split evenly: first half attend along H (V-stripes of T·H tokens + per column), second half attend along W (H-stripes of T·W tokens per + row). T is always full (matches Swin-3D and FullST conventions). + + State-dict layout intentionally identical to AxialAttention3D. + """ + + def __init__(self, dim: int, num_heads: int, + rope: Optional[RoPE3D] = None, + qkv_bias: bool = True, qk_norm: bool = True, + qkv_dw_kernel: int = 0): + super().__init__() + assert num_heads % 2 == 0, ( + f"CSwinAttention3D requires num_heads % 2 == 0, got {num_heads}") + self.dim = dim + self.num_heads = num_heads + self.h_heads = num_heads // 2 + self.w_heads = num_heads - self.h_heads + self.head_dim = dim // num_heads + self.qkv = _QKVProj(dim, num_heads, qkv_bias=qkv_bias) + self.qkv_dw = ( + QKVDepthwiseSpatial(dim, kernel_size=int(qkv_dw_kernel)) + if qkv_dw_kernel and qkv_dw_kernel > 0 else None + ) + self.attn_gate = HeadwiseAttentionGate(dim, num_heads) + self.qk_norm = qk_norm + if qk_norm: + self.q_norm = RMSNorm(self.head_dim) + self.k_norm = RMSNorm(self.head_dim) + self.proj = nn.Linear(dim, dim) + nn.init.zeros_(self.proj.weight) + if self.proj.bias is not None: + nn.init.zeros_(self.proj.bias) + self.rope = rope or RoPE3D(self.head_dim) + + def forward(self, x: torch.Tensor, t_stride: float = 1.0) -> torch.Tensor: + """x: (B, T, H, W, C). Returns (B, T, H, W, C).""" + B, T, H, W, C = x.shape + h_heads, w_heads, D = self.h_heads, self.w_heads, self.head_dim + strip_batch = int(getattr(self, 'inference_strip_batch', 0) or 0) + + if (strip_batch > 0 and not self.training and self.qkv_dw is None + and max(B * H, B * W) > strip_batch): + cos_h, sin_h = self.rope.get_cos_sin( + T, H, 1, device=x.device, t_stride=t_stride) + cos_w, sin_w = self.rope.get_cos_sin( + T, 1, W, device=x.device, t_stride=t_stride) + + x_h = x.permute(0, 3, 1, 2, 4).reshape(B * W, T * H, C) + out_h = x.new_empty((B * W, h_heads, T * H, D)) + for start in range(0, B * W, strip_batch): + end = min(start + strip_batch, B * W) + q, k, v = self.qkv(x_h[start:end]) + q = q[:, :h_heads] + k = k[:, :h_heads] + v = v[:, :h_heads] + if self.qk_norm: + q = self.q_norm(q) + k = self.k_norm(k) + q = apply_rotary(q, cos_h, sin_h) + k = apply_rotary(k, cos_h, sin_h) + out_h[start:end] = F.scaled_dot_product_attention( + q, k, v, dropout_p=0.0, is_causal=False) + attn_h = (out_h.reshape(B, W, h_heads, T, H, D) + .permute(0, 3, 4, 1, 2, 5) + .contiguous()) + + x_w = x.permute(0, 2, 1, 3, 4).reshape(B * H, T * W, C) + out_w = x.new_empty((B * H, w_heads, T * W, D)) + for start in range(0, B * H, strip_batch): + end = min(start + strip_batch, B * H) + q, k, v = self.qkv(x_w[start:end]) + q = q[:, h_heads:] + k = k[:, h_heads:] + v = v[:, h_heads:] + if self.qk_norm: + q = self.q_norm(q) + k = self.k_norm(k) + q = apply_rotary(q, cos_w, sin_w) + k = apply_rotary(k, cos_w, sin_w) + out_w[start:end] = F.scaled_dot_product_attention( + q, k, v, dropout_p=0.0, is_causal=False) + attn_w = (out_w.reshape(B, H, w_heads, T, W, D) + .permute(0, 3, 1, 4, 2, 5) + .contiguous()) + + attn = torch.cat([attn_h, attn_w], dim=-2) + attn = attn * self.attn_gate.grid(x, dtype=attn.dtype) + attn = attn.reshape(B, T, H, W, C) + return self.proj(attn) + + if self.qkv_dw is not None: + y = self.qkv_dw(self.qkv.qkv(x)) + qkv = (y.reshape(B, T * H * W, 3, self.num_heads, D) + .permute(2, 0, 3, 1, 4) + .contiguous()) + q, k, v = qkv.unbind(dim=0) + else: + x_flat = x.reshape(B, T * H * W, C) + q, k, v = self.qkv(x_flat) # (B, num_heads, T*H*W, D) + if self.qk_norm: + q = self.q_norm(q) + k = self.k_norm(k) + + q_h, k_h, v_h = q[:, :h_heads], k[:, :h_heads], v[:, :h_heads] + q_w, k_w, v_w = q[:, h_heads:], k[:, h_heads:], v[:, h_heads:] + + # H-group (V-stripes, T·H tokens per column): (B, h_heads, T*H*W, D) + # → (B, h_heads, T, H, W, D) → (B, W, h_heads, T, H, D) → (B·W, h_heads, T·H, D) + def _to_hstrip(t): + return (t.reshape(B, h_heads, T, H, W, D) + .permute(0, 4, 1, 2, 3, 5) + .reshape(B * W, h_heads, T * H, D)) + q_h, k_h, v_h = _to_hstrip(q_h), _to_hstrip(k_h), _to_hstrip(v_h) + + # W-group (H-stripes, T·W tokens per row): (B, w_heads, T*H*W, D) + # → (B, w_heads, T, H, W, D) → (B, H, w_heads, T, W, D) → (B·H, w_heads, T·W, D) + def _to_wstrip(t): + return (t.reshape(B, w_heads, T, H, W, D) + .permute(0, 3, 1, 2, 4, 5) + .reshape(B * H, w_heads, T * W, D)) + q_w, k_w, v_w = _to_wstrip(q_w), _to_wstrip(k_w), _to_wstrip(v_w) + + cos_h, sin_h = self.rope.get_cos_sin(T, H, 1, + device=x.device, t_stride=t_stride) + cos_w, sin_w = self.rope.get_cos_sin(T, 1, W, + device=x.device, t_stride=t_stride) + q_h = apply_rotary(q_h, cos_h, sin_h) + k_h = apply_rotary(k_h, cos_h, sin_h) + q_w = apply_rotary(q_w, cos_w, sin_w) + k_w = apply_rotary(k_w, cos_w, sin_w) + + attn_h = F.scaled_dot_product_attention(q_h, k_h, v_h, + dropout_p=0.0, is_causal=False) + attn_w = F.scaled_dot_product_attention(q_w, k_w, v_w, + dropout_p=0.0, is_causal=False) + + # Un-strip: H-group → (B, T, H, W, h_heads, D) + attn_h = (attn_h.reshape(B, W, h_heads, T, H, D) + .permute(0, 3, 4, 1, 2, 5) + .contiguous()) + # W-group → (B, T, H, W, w_heads, D) + attn_w = (attn_w.reshape(B, H, w_heads, T, W, D) + .permute(0, 3, 1, 4, 2, 5) + .contiguous()) + + attn = torch.cat([attn_h, attn_w], dim=-2) + attn = attn * self.attn_gate.grid(x, dtype=attn.dtype) + attn = attn.reshape(B, T, H, W, C) + return self.proj(attn) + + +class FullAttention3D(nn.Module): + """Global 3D attention (no windowing) with 3D RoPE. + + Used at /64 where the token count (~32² × T) is small enough for + quadratic attention to be cheap. + """ + + def __init__(self, dim: int, num_heads: int, + rope: Optional[RoPE3D] = None, + qkv_bias: bool = True, + qk_norm: bool = True, + qkv_dw_kernel: int = 0): + super().__init__() + self.dim = dim + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.qkv = _QKVProj(dim, num_heads, qkv_bias=qkv_bias) + self.qkv_dw = ( + QKVDepthwiseSpatial(dim, kernel_size=int(qkv_dw_kernel)) + if qkv_dw_kernel and qkv_dw_kernel > 0 else None + ) + self.attn_gate = HeadwiseAttentionGate(dim, num_heads) + self.qk_norm = qk_norm + if qk_norm: + self.q_norm = RMSNorm(self.head_dim) + self.k_norm = RMSNorm(self.head_dim) + self.proj = nn.Linear(dim, dim) + # Zero-init output proj so residual branch contributes 0 at init. + nn.init.zeros_(self.proj.weight) + if self.proj.bias is not None: + nn.init.zeros_(self.proj.bias) + self.rope = rope or RoPE3D(self.head_dim) + + def forward(self, x: torch.Tensor, t_stride: float = 1.0) -> torch.Tensor: + """x: (B, T, H, W, C). Returns (B, T, H, W, C).""" + B, T, H, W, C = x.shape + N = T * H * W + query_chunk = int(getattr(self, 'inference_query_chunk_tokens', 0) or 0) + if self.qkv_dw is not None: + y = self.qkv_dw(self.qkv.qkv(x)) + qkv = (y.reshape(B, N, 3, self.num_heads, self.head_dim) + .permute(2, 0, 3, 1, 4) + .contiguous()) + q, k, v = qkv.unbind(dim=0) + else: + seq = x.reshape(B, N, C) + q, k, v = self.qkv(seq) + gate_tokens = x.reshape(B, N, C) + if self.qk_norm: + q = self.q_norm(q) + k = self.k_norm(k) + cos, sin = self.rope.get_cos_sin(T, H, W, device=x.device, + t_stride=t_stride) + q = apply_rotary(q, cos, sin) + k = apply_rotary(k, cos, sin) + if query_chunk > 0 and not self.training and N > query_chunk: + attn = q.new_empty((B, self.num_heads, N, self.head_dim)) + for start in range(0, N, query_chunk): + end = min(start + query_chunk, N) + a = F.scaled_dot_product_attention( + q[:, :, start:end], k, v, + dropout_p=0.0, is_causal=False) + a = self.attn_gate(gate_tokens[:, start:end], a) + attn[:, :, start:end] = a + else: + attn = F.scaled_dot_product_attention( + q, k, v, dropout_p=0.0, is_causal=False) + attn = self.attn_gate(gate_tokens, attn) + attn = attn.transpose(1, 2).reshape(B, N, C) + attn = self.proj(attn) + return attn.reshape(B, T, H, W, C) + + +class FullResolutionRecurrentMemorySite(nn.Module): + """Full-/64 recurrent memory site. + + The persistent state has the same H×W grid and channel width as the /64 + stream. Each frame first reads from the accumulated state via full + cross-attention, then writes the current representation back with an RVM- + style gated transformer candidate. Unlike RVM, the candidate updates only + the recurrent memory; the frame stream remains the current-frame features + plus the explicit read residual. The read output projection is zero- + initialized, so adding the site to an existing checkpoint is behavior- + neutral. + """ + + def __init__(self, dim: int, num_heads: int, + rope: Optional[RoPE2D] = None, + qkv_bias: bool = True, + qk_norm: bool = True): + super().__init__() + if dim % num_heads != 0: + raise ValueError( + f"dim={dim} must divide num_heads={num_heads}") + self.dim = int(dim) + self.num_heads = int(num_heads) + self.head_dim = self.dim // self.num_heads + + self.norm_x = RMSNorm(dim) + self.norm_state = RMSNorm(dim) + self.q = nn.Linear(dim, dim, bias=qkv_bias) + self.k = nn.Linear(dim, dim, bias=qkv_bias) + self.v = nn.Linear(dim, dim, bias=qkv_bias) + self.attn_gate = HeadwiseAttentionGate(dim, num_heads) + self.qk_norm = bool(qk_norm) + if self.qk_norm: + self.q_norm = RMSNorm(self.head_dim) + self.k_norm = RMSNorm(self.head_dim) + self.proj = nn.Linear(dim, dim) + nn.init.zeros_(self.proj.weight) + if self.proj.bias is not None: + nn.init.zeros_(self.proj.bias) + self.rope = rope or RoPE2D(self.head_dim) + + self.write_norm_state = RMSNorm(dim) + self.input_update = nn.Linear(dim, dim, bias=False) + self.state_update = nn.Linear(dim, dim, bias=False) + self.input_reset = nn.Linear(dim, dim, bias=False) + self.state_reset = nn.Linear(dim, dim, bias=False) + nn.init.zeros_(self.input_update.weight) + nn.init.zeros_(self.state_update.weight) + nn.init.zeros_(self.input_reset.weight) + nn.init.zeros_(self.state_reset.weight) + + self.write_ca_norm = RMSNorm(dim) + self.write_ca_q = nn.Linear(dim, dim, bias=qkv_bias) + self.write_ca_k = nn.Linear(dim, dim, bias=qkv_bias) + self.write_ca_v = nn.Linear(dim, dim, bias=qkv_bias) + self.write_ca_gate = HeadwiseAttentionGate(dim, num_heads) + if self.qk_norm: + self.write_ca_q_norm = RMSNorm(self.head_dim) + self.write_ca_k_norm = RMSNorm(self.head_dim) + self.write_ca_proj = nn.Linear(dim, dim) + nn.init.zeros_(self.write_ca_proj.weight) + if self.write_ca_proj.bias is not None: + nn.init.zeros_(self.write_ca_proj.bias) + + def _split_heads(self, tokens: torch.Tensor, + proj: nn.Linear) -> torch.Tensor: + B, N, C = tokens.shape + return proj(tokens).reshape( + B, N, self.num_heads, self.head_dim).transpose(1, 2) + + def _read(self, x_t: torch.Tensor, state: torch.Tensor) -> torch.Tensor: + B, H, W, C = x_t.shape + if tuple(state.shape) != (B, H, W, C): + raise ValueError( + f"recurrent state shape {tuple(state.shape)} != " + f"expected {(B, H, W, C)}") + N = H * W + x_norm = self.norm_x(x_t) + s_norm = self.norm_state(state) + q_tokens = x_norm.reshape(B, N, C) + k_tokens = s_norm.reshape(B, N, C) + q = self._split_heads(q_tokens, self.q) + k = self._split_heads(k_tokens, self.k) + v = self._split_heads(k_tokens, self.v) + if self.qk_norm: + q = self.q_norm(q) + k = self.k_norm(k) + cos, sin = self.rope.get_cos_sin(H, W, device=x_t.device) + q = apply_rotary(q, cos, sin) + k = apply_rotary(k, cos, sin) + attn = F.scaled_dot_product_attention( + q, k, v, dropout_p=0.0, is_causal=False) + attn = self.attn_gate(q_tokens, attn) + attn = attn.transpose(1, 2).reshape(B, N, C) + read = self.proj(attn).reshape(B, H, W, C) + return x_t + read + + def _write_cross_attention( + self, + query_grid: torch.Tensor, + kv_grid: torch.Tensor, + ) -> torch.Tensor: + B, H, W, C = query_grid.shape + N = H * W + q_tokens = self.write_ca_norm(query_grid).reshape(B, N, C) + kv_tokens = kv_grid.reshape(B, N, C) + q = self._split_heads(q_tokens, self.write_ca_q) + k = self._split_heads(kv_tokens, self.write_ca_k) + v = self._split_heads(kv_tokens, self.write_ca_v) + if self.qk_norm: + q = self.write_ca_q_norm(q) + k = self.write_ca_k_norm(k) + cos, sin = self.rope.get_cos_sin(H, W, device=query_grid.device) + q = apply_rotary(q, cos, sin) + k = apply_rotary(k, cos, sin) + attn = F.scaled_dot_product_attention( + q, k, v, dropout_p=0.0, is_causal=False) + attn = self.write_ca_gate(q_tokens, attn) + attn = attn.transpose(1, 2).reshape(B, N, C) + attn = self.write_ca_proj(attn).reshape(B, H, W, C) + return attn + + def _write_candidate( + self, + x_t: torch.Tensor, + gated_state: torch.Tensor, + ) -> torch.Tensor: + return x_t + self._write_cross_attention(x_t, gated_state) + + def _write(self, x_t: torch.Tensor, state: torch.Tensor) -> torch.Tensor: + update = torch.sigmoid(self.input_update(x_t) + + self.state_update(state)) + reset = torch.sigmoid(self.input_reset(x_t) + + self.state_reset(state)) + gated_state = reset * self.write_norm_state(state) + candidate = self._write_candidate(x_t, gated_state) + return state + update * (candidate - state) + + def forward( + self, + x: torch.Tensor, + state: torch.Tensor | None = None, + ) -> tuple[torch.Tensor, torch.Tensor]: + """x: (B, T, H, W, C); state: (B, H, W, C) or None.""" + B, T, H, W, C = x.shape + if C != self.dim: + raise ValueError(f"input channels {C} != recurrent dim {self.dim}") + if state is None: + state = x[:, 0].clone() + elif tuple(state.shape) != (B, H, W, C): + raise ValueError( + f"recurrent state shape {tuple(state.shape)} != " + f"expected {(B, H, W, C)}") + outs = [] + for t in range(T): + x_t = self._read(x[:, t], state) + state = self._write(x_t, state) + outs.append(x_t) + return torch.stack(outs, dim=1), state + + +class FullSpatialAttention2D(nn.Module): + """Per-frame full-spatial attention for NTHWC video tensors. + + Attention is global over H×W independently for each frame; there is no + temporal mixing here. This is used before the /16 → /64 downsample so the + high-resolution local/hint features can exchange spatial information while + leaving temporal/global video reasoning to the /64 body. + """ + + def __init__(self, dim: int, num_heads: int, + rope: Optional[RoPE2D] = None, + qkv_bias: bool = True, + qk_norm: bool = True, + zero_init_proj: bool = True, + qkv_dw_kernel: int = 0): + super().__init__() + self.dim = dim + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.qkv = _QKVProj(dim, num_heads, qkv_bias=qkv_bias) + self.qkv_dw = ( + QKVDepthwiseSpatial(dim, kernel_size=int(qkv_dw_kernel)) + if qkv_dw_kernel and qkv_dw_kernel > 0 else None + ) + self.attn_gate = HeadwiseAttentionGate(dim, num_heads) + self.qk_norm = qk_norm + if qk_norm: + self.q_norm = RMSNorm(self.head_dim) + self.k_norm = RMSNorm(self.head_dim) + self.proj = nn.Linear(dim, dim) + if zero_init_proj: + nn.init.zeros_(self.proj.weight) + if self.proj.bias is not None: + nn.init.zeros_(self.proj.bias) + self.rope = rope or RoPE2D(self.head_dim) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """x: (B, T, H, W, C). Returns same shape.""" + B, T, H, W, C = x.shape + N = H * W + BT = B * T + query_chunk = int(getattr(self, 'inference_query_chunk_tokens', 0) or 0) + if self.qkv_dw is not None: + y = self.qkv_dw(self.qkv.qkv(x)) + qkv = (y.reshape(BT, N, 3, self.num_heads, self.head_dim) + .permute(2, 0, 3, 1, 4) + .contiguous()) + q, k, v = qkv.unbind(dim=0) + else: + tokens = x.reshape(BT, N, C) + q, k, v = self.qkv(tokens) + if self.qk_norm: + q = self.q_norm(q) + k = self.k_norm(k) + cos, sin = self.rope.get_cos_sin(H, W, device=x.device) + q = apply_rotary(q, cos, sin) + k = apply_rotary(k, cos, sin) + + gate_tokens = x.reshape(BT, N, C) + if query_chunk > 0 and not self.training and N > query_chunk: + attn = q.new_empty((BT, self.num_heads, N, self.head_dim)) + for start in range(0, N, query_chunk): + end = min(start + query_chunk, N) + a = F.scaled_dot_product_attention( + q[:, :, start:end], k, v, + dropout_p=0.0, is_causal=False) + a = self.attn_gate(gate_tokens[:, start:end], a) + attn[:, :, start:end] = a + else: + attn = F.scaled_dot_product_attention( + q, k, v, dropout_p=0.0, is_causal=False) + attn = self.attn_gate(gate_tokens, attn) + attn = attn.transpose(1, 2).reshape(BT, N, C) + attn = self.proj(attn) + return attn.reshape(B, T, H, W, C) + + +class FullSpatialBlock2D(nn.Module): + """Pre-norm block with full H×W attention for BHWC tensors. + + Used sparingly in stage 2 to provide global spatial exchange while the + surrounding Swin/CSwin blocks keep the cheaper local/axial inductive bias. + """ + + def __init__(self, dim: int, num_heads: int, + mlp_mult: float = 2.5, + rope: Optional[RoPE2D] = None, + local_mix_kernel: int = 0, + local_mix_init_std: float | None = None, + mlp_dw_kernel: int = 0, + qkv_dw_kernel: int = 0): + super().__init__() + self.dim = dim + self.norm1 = RMSNorm(dim) + self.attn = FullSpatialAttention2D( + dim, num_heads, rope=rope, + qkv_dw_kernel=qkv_dw_kernel, + ) + self.local_mix = ( + DepthwiseSwiGLU2D(dim, kernel_size=local_mix_kernel, + init_std=local_mix_init_std) + if local_mix_kernel and local_mix_kernel > 0 else None + ) + self.norm2 = RMSNorm(dim) + self.mlp = SwiGLU(dim, mult=mlp_mult, dw_kernel=mlp_dw_kernel) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """x: (B, H, W, C). Returns same shape.""" + shortcut = x + x = self.norm1(x) + x = self.attn(x[:, None]).squeeze(1) + x = shortcut + x + + if self.local_mix is not None: + x = x + self.local_mix(x) + + shortcut = x + x = self.norm2(x) + x = self.mlp(x) + x = shortcut + x + return x + + +# --------------------------------------------------------------------------- +# LayerScale — grows residual contribution from ~0 at init +# --------------------------------------------------------------------------- + + +class LayerScale(nn.Module): + """Per-channel scalar gamma multiplier, init=1e-6 (near-identity residual).""" + + def __init__(self, dim: int, init_value: float = 1e-6): + super().__init__() + self.gamma = nn.Parameter(init_value * torch.ones(dim)) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return x * self.gamma + + +# --------------------------------------------------------------------------- +# Block wrappers (Swin-2D, Swin-3D, Full-3D) +# --------------------------------------------------------------------------- + + +def _cyclic_shift_2d(x: torch.Tensor, shift: Tuple[int, int]) -> torch.Tensor: + """Roll a 2D spatial grid by (h, w). Caller provides the sign.""" + sh, sw = shift + if sh == 0 and sw == 0: + return x + # x is (B, H, W, C) + return torch.roll(x, shifts=(sh, sw), dims=(1, 2)) + + +def _cyclic_shift_3d(x: torch.Tensor, shift: Tuple[int, int, int]) -> torch.Tensor: + """Roll a 3D spatiotemporal grid by (t, h, w).""" + st, sh, sw = shift + if st == 0 and sh == 0 and sw == 0: + return x + # x is (B, T, H, W, C) + return torch.roll(x, shifts=(st, sh, sw), dims=(1, 2, 3)) + + +def _resolve_shift_2d(shift, window_size: int) -> Tuple[int, int]: + """Accept shift as bool (legacy) or (sh, sw) tuple. Bool True → (w/2, w/2).""" + if shift is True: + return (window_size // 2, window_size // 2) + if shift is False or shift is None: + return (0, 0) + assert isinstance(shift, (tuple, list)) and len(shift) == 2, ( + f"shift must be bool or (sh, sw); got {shift!r}") + return (int(shift[0]), int(shift[1])) + + +def _resolve_shift_3d(shift, window_H: int, window_W: int) -> Tuple[int, int, int]: + """Accept shift as bool (legacy) or (sh, sw) tuple for SPATIAL shift. + Temporal shift is never used (T_w = T). Returns (st=0, sh, sw).""" + if shift is True: + return (0, window_H // 2, window_W // 2) + if shift is False or shift is None: + return (0, 0, 0) + assert isinstance(shift, (tuple, list)) and len(shift) == 2, ( + f"shift must be bool or (sh, sw); got {shift!r}") + return (0, int(shift[0]), int(shift[1])) + + +class SwinBlock2D(nn.Module): + """Pre-norm Swin-2D block: RMSNorm → (shift?) → WindowAttn2D → residual + → optional DW local mix → RMSNorm → SwiGLU → residual. + + `shift` accepts bool (legacy: True → (w/2, w/2)) OR a `(sh, sw)` tuple + for the 4-phase pattern: (0, 0), (w/2, w/2), (w/2, 0), (0, w/2). + + Near-identity at init via zero-init output projections (attn.proj + + SwiGLU.w_down). No LayerScale. + """ + + def __init__(self, dim: int, num_heads: int, window_size: int = 8, + shift=False, mlp_mult: float = 2.5, + rope: Optional[RoPE2D] = None, + local_mix_kernel: int = 0, + local_mix_init_std: float | None = None, + mlp_dw_kernel: int = 0, + qkv_dw_kernel: int = 0): + super().__init__() + self.dim = dim + self.window_size = window_size + self.shift_h, self.shift_w = _resolve_shift_2d(shift, window_size) + self.shift = (self.shift_h, self.shift_w) + + self.norm1 = RMSNorm(dim) + self.attn = SwinAttention2D( + dim, num_heads, window_size, rope=rope, + qkv_dw_kernel=qkv_dw_kernel, + ) + + self.local_mix = ( + DepthwiseSwiGLU2D(dim, kernel_size=local_mix_kernel, + init_std=local_mix_init_std) + if local_mix_kernel and local_mix_kernel > 0 else None + ) + + self.norm2 = RMSNorm(dim) + self.mlp = SwiGLU(dim, mult=mlp_mult, dw_kernel=mlp_dw_kernel) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """x: (B, H, W, C). Returns same shape.""" + sh, sw = self.shift_h, self.shift_w + shortcut = x + x = self.norm1(x) + if sh or sw: + H, W = x.shape[1], x.shape[2] + w = self.window_size + Hp = math.ceil(H / w) * w + Wp = math.ceil(W / w) * w + if Hp != H or Wp != W: + x = F.pad(x, (0, 0, 0, Wp - W, 0, Hp - H)) + region_groups = _shifted_region_groups_2d_cached( + H, W, Hp, Wp, w, sh, sw) + x = _cyclic_shift_2d(x, (-sh, -sw)) + x = self.attn(x, region_groups=region_groups) + x = _cyclic_shift_2d(x, (sh, sw)) + x = x[:, :H, :W, :].contiguous() + else: + x = self.attn(x) + x = shortcut + x + + if self.local_mix is not None: + x = x + self.local_mix(x) + + shortcut = x + x = self.norm2(x) + x = self.mlp(x) + x = shortcut + x + return x + + +class AxialBlock2D(nn.Module): + """Pre-norm axial 2D block: RMSNorm → AxialAttention2D(axis) → residual + → RMSNorm → SwiGLU → residual. No padding, no windowing.""" + + def __init__(self, dim: int, num_heads: int, axis: str, + mlp_mult: float = 2.5, + rope: Optional[RoPE2D] = None): + super().__init__() + self.dim = dim + self.axis = axis + self.norm1 = RMSNorm(dim) + self.attn = AxialAttention2D(dim, num_heads, axis=axis, rope=rope) + self.norm2 = RMSNorm(dim) + self.mlp = SwiGLU(dim, mult=mlp_mult) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + shortcut = x + x = self.norm1(x) + x = self.attn(x) + x = shortcut + x + shortcut = x + x = self.norm2(x) + x = self.mlp(x) + x = shortcut + x + return x + + +class CSwinBlock2D(nn.Module): + """Pre-norm CSwin 2D block: RMSNorm → CSwinAttention2D (half heads on + H-axis, half on W-axis) → residual → optional DW local mix → RMSNorm + → SwiGLU → residual. + + Each block yields a globally-mixed output on both axes in a single + attention call. + """ + + def __init__(self, dim: int, num_heads: int, + mlp_mult: float = 2.5, + rope: Optional[RoPE2D] = None, + local_mix_kernel: int = 0, + local_mix_init_std: float | None = None, + mlp_dw_kernel: int = 0, + qkv_dw_kernel: int = 0): + super().__init__() + self.dim = dim + self.norm1 = RMSNorm(dim) + self.attn = CSwinAttention2D( + dim, num_heads, rope=rope, + qkv_dw_kernel=qkv_dw_kernel, + ) + self.local_mix = ( + DepthwiseSwiGLU2D(dim, kernel_size=local_mix_kernel, + init_std=local_mix_init_std) + if local_mix_kernel and local_mix_kernel > 0 else None + ) + self.norm2 = RMSNorm(dim) + self.mlp = SwiGLU(dim, mult=mlp_mult, dw_kernel=mlp_dw_kernel) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + shortcut = x + x = self.norm1(x) + x = self.attn(x) + x = shortcut + x + if self.local_mix is not None: + x = x + self.local_mix(x) + shortcut = x + x = self.norm2(x) + x = self.mlp(x) + x = shortcut + x + return x + + +class FullSpatialBlock3D(nn.Module): + """Pre-norm per-frame full-spatial block for stage-1 pre-downsample use. + + RMSNorm → full H×W attention per frame → residual → optional per-frame DW + local mix → RMSNorm → SwiGLU → residual. Temporal mixing is deliberately + absent; the /64 body handles full spatiotemporal reasoning after downsample. + """ + + def __init__(self, dim: int, num_heads: int, + mlp_mult: float = 2.5, + rope: Optional[RoPE2D] = None, + local_mix_kernel: int = 0, + local_mix_init_std: float | None = None, + adaln_cond_dim: int = 0, + mlp_dw_kernel: int = 0, + qkv_dw_kernel: int = 0): + super().__init__() + self.dim = dim + self.use_adaln_zero = bool(adaln_cond_dim and adaln_cond_dim > 0) + self.norm1 = RMSNorm(dim) + self.attn = FullSpatialAttention2D( + dim, num_heads, rope=rope, + zero_init_proj=not self.use_adaln_zero, + qkv_dw_kernel=qkv_dw_kernel, + ) + self.local_mix = ( + DepthwiseSwiGLU2D(dim, kernel_size=local_mix_kernel, + init_std=local_mix_init_std) + if local_mix_kernel and local_mix_kernel > 0 else None + ) + self.adaln = ( + AdaLNZero(dim, cond_dim=adaln_cond_dim, pairs=2) + if self.use_adaln_zero else None + ) + self.norm2 = RMSNorm(dim) + self.mlp = SwiGLU(dim, mult=mlp_mult, + zero_init_down=not self.use_adaln_zero, + dw_kernel=mlp_dw_kernel) + + def _local_mix_frames(self, x: torch.Tensor) -> torch.Tensor: + if self.local_mix is None: + return x + B, T, H, W, C = x.shape + y = x.reshape(B * T, H, W, C) + y = self.local_mix(y) + return x + y.reshape(B, T, H, W, C) + + def forward( + self, + x: torch.Tensor, + t_stride: float = 1.0, + adaln_cond: torch.Tensor | None = None, + ) -> torch.Tensor: + del t_stride + if self.adaln is not None and adaln_cond is None: + adaln_cond = x.new_zeros(x.shape[0], x.shape[1]) + if self.adaln is not None: + shift1, scale1, gate1, shift2, scale2, gate2 = self.adaln(adaln_cond, x) + else: + shift1 = scale1 = gate1 = shift2 = scale2 = gate2 = None + + shortcut = x + x = self.norm1(x) + if shift1 is not None: + x = x * (1.0 + scale1) + shift1 + x = self.attn(x) + x = shortcut + (x * gate1 if gate1 is not None else x) + + x = self._local_mix_frames(x) + + shortcut = x + x = self.norm2(x) + if shift2 is not None: + x = x * (1.0 + scale2) + shift2 + x = self.mlp(x) + x = shortcut + (x * gate2 if gate2 is not None else x) + return x + + +class SwinBlock3D(nn.Module): + """Pre-norm Swin-3D block. Full-temporal + windowed-spatial attention + + optional per-frame DW local mix + SwiGLU. `shift` accepts bool + (legacy: True → (w/2, w/2)) OR a `(sh, sw)` tuple for 4-phase patterns: + (0,0), (w/2,w/2), (w/2,0), (0,w/2). Temporal shift is always 0 + (T_w = T). + + Near-identity at init via zero-init down projections (no LayerScale). + """ + + def __init__(self, dim: int, num_heads: int, + window_H: int = 8, window_W: int = 8, + shift=False, mlp_mult: float = 2.5, + rope: Optional[RoPE3D] = None, + local_mix_kernel: int = 0, + local_mix_init_std: float | None = None, + adaln_cond_dim: int = 0, + mlp_dw_kernel: int = 0, + qkv_dw_kernel: int = 0): + super().__init__() + self.dim = dim + self.window_H = window_H + self.window_W = window_W + _, self.shift_h, self.shift_w = _resolve_shift_3d(shift, window_H, window_W) + self.shift = (self.shift_h, self.shift_w) + self.use_adaln_zero = bool(adaln_cond_dim and adaln_cond_dim > 0) + + self.norm1 = RMSNorm(dim) + self.attn = SwinAttention3D( + dim, num_heads, window_H, window_W, rope=rope, + zero_init_proj=not self.use_adaln_zero, + qkv_dw_kernel=qkv_dw_kernel, + ) + + self.local_mix = ( + DepthwiseSwiGLU2D(dim, kernel_size=local_mix_kernel, + init_std=local_mix_init_std) + if local_mix_kernel and local_mix_kernel > 0 else None + ) + self.adaln = ( + AdaLNZero(dim, cond_dim=adaln_cond_dim, pairs=2) + if self.use_adaln_zero else None + ) + + self.norm2 = RMSNorm(dim) + self.mlp = SwiGLU(dim, mult=mlp_mult, + zero_init_down=not self.use_adaln_zero, + dw_kernel=mlp_dw_kernel) + + def _local_mix_frames(self, x: torch.Tensor) -> torch.Tensor: + if self.local_mix is None: + return x + B, T, H, W, C = x.shape + y = x.reshape(B * T, H, W, C) + y = self.local_mix(y) + return x + y.reshape(B, T, H, W, C) + + def forward( + self, + x: torch.Tensor, + t_stride: float = 1.0, + adaln_cond: torch.Tensor | None = None, + ) -> torch.Tensor: + """x: (B, T, H, W, C). Returns (B, T, H, W, C).""" + if self.adaln is not None and adaln_cond is None: + adaln_cond = x.new_zeros(x.shape[0], x.shape[1]) + if self.adaln is not None: + shift1, scale1, gate1, shift2, scale2, gate2 = self.adaln(adaln_cond, x) + else: + shift1 = scale1 = gate1 = shift2 = scale2 = gate2 = None + sh, sw = self.shift_h, self.shift_w + shortcut = x + x = self.norm1(x) + if shift1 is not None: + x = x * (1.0 + scale1) + shift1 + if sh or sw: + T, H, W = x.shape[1], x.shape[2], x.shape[3] + H_w, W_w = self.window_H, self.window_W + Hp = math.ceil(H / H_w) * H_w + Wp = math.ceil(W / W_w) * W_w + if Hp != H or Wp != W: + x = F.pad(x, (0, 0, 0, Wp - W, 0, Hp - H)) + region_groups = _shifted_region_groups_3d_cached( + T, H, W, Hp, Wp, H_w, W_w, sh, sw) + x = _cyclic_shift_3d(x, (0, -sh, -sw)) + x = self.attn(x, t_stride=t_stride, region_groups=region_groups) + x = _cyclic_shift_3d(x, (0, sh, sw)) + x = x[:, :, :H, :W, :].contiguous() + else: + x = self.attn(x, t_stride=t_stride) + x = shortcut + (x * gate1 if gate1 is not None else x) + + x = self._local_mix_frames(x) + + shortcut = x + x = self.norm2(x) + if shift2 is not None: + x = x * (1.0 + scale2) + shift2 + x = self.mlp(x) + x = shortcut + (x * gate2 if gate2 is not None else x) + return x + + +class AxialBlock3D(nn.Module): + """Pre-norm axial 3D block: RMSNorm → AxialAttention3D(axis) → residual + → RMSNorm → SwiGLU → residual. T-full + one spatial axis.""" + + def __init__(self, dim: int, num_heads: int, axis: str, + mlp_mult: float = 2.5, + rope: Optional[RoPE3D] = None): + super().__init__() + self.dim = dim + self.axis = axis + self.norm1 = RMSNorm(dim) + self.attn = AxialAttention3D(dim, num_heads, axis=axis, rope=rope) + self.norm2 = RMSNorm(dim) + self.mlp = SwiGLU(dim, mult=mlp_mult) + + def forward(self, x: torch.Tensor, t_stride: float = 1.0) -> torch.Tensor: + shortcut = x + x = self.norm1(x) + x = self.attn(x, t_stride=t_stride) + x = shortcut + x + shortcut = x + x = self.norm2(x) + x = self.mlp(x) + x = shortcut + x + return x + + +class CSwinBlock3D(nn.Module): + """Pre-norm CSwin 3D block: RMSNorm → CSwinAttention3D (half heads on + H-axis, half on W-axis; T always full) → residual → optional per-frame + DW local mix → RMSNorm → SwiGLU → residual. + """ + + def __init__(self, dim: int, num_heads: int, + mlp_mult: float = 2.5, + rope: Optional[RoPE3D] = None, + local_mix_kernel: int = 0, + local_mix_init_std: float | None = None, + mlp_dw_kernel: int = 0, + qkv_dw_kernel: int = 0): + super().__init__() + self.dim = dim + self.norm1 = RMSNorm(dim) + self.attn = CSwinAttention3D( + dim, num_heads, rope=rope, + qkv_dw_kernel=qkv_dw_kernel, + ) + self.local_mix = ( + DepthwiseSwiGLU2D(dim, kernel_size=local_mix_kernel, + init_std=local_mix_init_std) + if local_mix_kernel and local_mix_kernel > 0 else None + ) + self.norm2 = RMSNorm(dim) + self.mlp = SwiGLU(dim, mult=mlp_mult, dw_kernel=mlp_dw_kernel) + + def _local_mix_frames(self, x: torch.Tensor) -> torch.Tensor: + if self.local_mix is None: + return x + B, T, H, W, C = x.shape + y = x.reshape(B * T, H, W, C) + y = self.local_mix(y) + return x + y.reshape(B, T, H, W, C) + + def forward(self, x: torch.Tensor, t_stride: float = 1.0) -> torch.Tensor: + shortcut = x + x = self.norm1(x) + x = self.attn(x, t_stride=t_stride) + x = shortcut + x + x = self._local_mix_frames(x) + shortcut = x + x = self.norm2(x) + x = self.mlp(x) + x = shortcut + x + return x + + +class FullSpatioTemporalBlock(nn.Module): + """Pre-norm block with full (unwindowed) 3D attention. For /64 body. + Near-identity at init via zero-init down projections (no LayerScale). + """ + + def __init__(self, dim: int, num_heads: int, + mlp_mult: float = 2.5, + rope: Optional[RoPE3D] = None, + local_mix_kernel: int = 0, + local_mix_init_std: float | None = None, + mlp_dw_kernel: int = 0, + mlp_hidden: int | None = None, + qkv_dw_kernel: int = 0): + super().__init__() + self.norm1 = RMSNorm(dim) + self.attn = FullAttention3D( + dim, num_heads, rope=rope, + qkv_dw_kernel=qkv_dw_kernel, + ) + self.local_mix = ( + DepthwiseSwiGLU2D(dim, kernel_size=local_mix_kernel, + init_std=local_mix_init_std) + if local_mix_kernel and local_mix_kernel > 0 else None + ) + self.norm2 = RMSNorm(dim) + self.mlp = SwiGLU(dim, mult=mlp_mult, dw_kernel=mlp_dw_kernel, + hidden=mlp_hidden) + + def _local_mix_frames(self, x: torch.Tensor) -> torch.Tensor: + if self.local_mix is None: + return x + B, T, H, W, C = x.shape + y = x.reshape(B * T, H, W, C) + y = self.local_mix(y) + return x + y.reshape(B, T, H, W, C) + + def forward( + self, + x: torch.Tensor, + t_stride: float = 1.0, + recurrent_site: FullResolutionRecurrentMemorySite | None = None, + recurrent_state: torch.Tensor | None = None, + ) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]: + """x: (B, T, H, W, C). Returns (B, T, H, W, C).""" + shortcut = x + x = self.norm1(x) + x = self.attn(x, t_stride=t_stride) + x = shortcut + x + x = self._local_mix_frames(x) + if recurrent_site is not None: + x, recurrent_state = recurrent_site(x, recurrent_state) + + shortcut = x + x = self.norm2(x) + x = self.mlp(x) + x = shortcut + x + if recurrent_site is not None: + return x, recurrent_state + return x + + +# --------------------------------------------------------------------------- +# Patchify / downsample / upsample / depatchify +# --------------------------------------------------------------------------- + + +class PatchifyConv(nn.Module): + """Conv2d(kernel=r, stride=r) that turns (B, C_in, H, W) into + (B, C_out, H/r, W/r). Supports partial zero-init on output channels via + `zero_init_first` — the first N output channels have their weights set + to zero while the rest get Kaiming. + + `overlap_mult` controls the receptive field size: + • 1 (default): kernel=patch, stride=patch — no overlap (pure ViT patch). + • 2: kernel=2·patch, stride=patch, padding=patch/2 — each token's RF + covers its own 16×16 + half-width strips of the 4 axis neighbors + + quarter-size corners of the 4 diagonal neighbors. ("Half-overlap") + • 3: kernel=3·patch, stride=patch, padding=patch — immediate + 8 + surrounding 16×16 blocks fully covered. ("Full-overlap") + + `overlap=True` is kept as an alias for overlap_mult=3. + + Init behavior for overlap_mult > 1: the CENTRAL patch×patch region of + each kernel is Kaiming-inited (producing the "own block" features + identically to the non-overlap baseline at step 0); the surrounding + spill regions that reach into neighboring 16×16 blocks are + ZERO-inited. This way at init a token is a pure function of its own + block (same as non-overlap), and the overlap contribution from + neighbors emerges only as those spill weights train away from zero. + """ + + def __init__(self, in_channels: int, out_channels: int, patch: int, + zero_init_first: int = 0, + zero_init_all: bool = False, + bias: bool = True, + overlap: bool = False, + overlap_mult: int = 1): + super().__init__() + self.patch = patch + if overlap and overlap_mult == 1: + overlap_mult = 3 # legacy alias + assert overlap_mult >= 1, f"overlap_mult must be >= 1, got {overlap_mult}" + self.overlap_mult = overlap_mult + kernel = overlap_mult * patch + padding = (kernel - patch) // 2 + assert (kernel - patch) % 2 == 0, ( + f"overlap_mult·patch - patch must be even; got kernel={kernel}, patch={patch}") + self.conv = nn.Conv2d(in_channels, out_channels, + kernel_size=kernel, stride=patch, + padding=padding, bias=bias) + if zero_init_all: + nn.init.zeros_(self.conv.weight) + if self.conv.bias is not None: + nn.init.zeros_(self.conv.bias) + else: + # Default nn.Conv2d __init__ already Kaiming-inited the full + # kernel + uniform-inited the bias. For overlap_mult > 1 we + # want the Kaiming signal only in the central patch×patch + # region; zero the spill. For `zero_init_first` we additionally + # zero the first N output channels entirely (kernel *and* bias + # for those channels). + with torch.no_grad(): + if overlap_mult > 1: + # Zero the spill: build a (kernel, kernel) mask with a + # patch×patch block of 1s in the center. + c_lo = padding + c_hi = padding + patch + mask = torch.zeros(kernel, kernel, + dtype=self.conv.weight.dtype, + device=self.conv.weight.device) + mask[c_lo:c_hi, c_lo:c_hi] = 1.0 + self.conv.weight.data.mul_(mask) + if zero_init_first > 0: + self.conv.weight.data[:zero_init_first].zero_() + if self.conv.bias is not None: + self.conv.bias.data[:zero_init_first].zero_() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.conv(x) + + +class Downsample4(nn.Module): + """Conv(k=8, s=4) downsample from /16 → /64. Kernel>stride gives overlapping + receptive fields (an implicit low-pass before sub-sample), an improvement + over the minimum-cost k=4 s=4 form. Output layer-norm by default.""" + + def __init__(self, in_channels: int, out_channels: int, + kernel: int = 8, stride: int = 4, norm: bool = True): + super().__init__() + assert kernel >= stride, "kernel smaller than stride would drop pixels" + # Padding chosen so the output size is exactly in//stride (for in % stride == 0). + # pad = (kernel - stride) // 2. At k=8 s=4 → pad=2 (centered). + pad = (kernel - stride) // 2 + self.conv = nn.Conv2d(in_channels, out_channels, + kernel_size=kernel, stride=stride, padding=pad) + self.norm = LayerNorm2d(out_channels) if norm else nn.Identity() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.norm(self.conv(x)) + + +class Upsample4ICNR(nn.Module): + """ICNR-inited /64 → /16 upsample: 1×1 projection to c_out·r² channels, + PixelShuffle(r=4), optional post Conv(k=3) smoothing. + + Post-conv is symmetric in spirit with the kernel-8 overlap used on + the downsample side — it mixes across the 4×4 pixel-shuffle output + at cost `c_out² · 9`. Disable via `post_smooth=False` if the next + transformer blocks are considered sufficient to smooth. + """ + + def __init__(self, in_channels: int, out_channels: int, + r: int = 4, post_smooth: bool = True, norm: bool = True): + super().__init__() + self.r = r + self.proj = nn.Conv2d(in_channels, out_channels * r * r, kernel_size=1) + icnr_init_(self.proj, upscale_factor=r) + self.post_smooth = post_smooth + if post_smooth: + self.smooth = nn.Conv2d(out_channels, out_channels, + kernel_size=3, padding=1) + # Zero-init so the upsample path is pure nearest-neighbor at t=0. + # Combined with a residual around smooth in forward, this + # preserves the ICNR pixel-shuffle output unchanged at init and + # lets the 3×3 smoother grow in as a learned correction. + nn.init.zeros_(self.smooth.weight) + if self.smooth.bias is not None: + nn.init.zeros_(self.smooth.bias) + else: + self.smooth = nn.Identity() + self.norm = LayerNorm2d(out_channels) if norm else nn.Identity() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = self.proj(x) + x = F.pixel_shuffle(x, self.r) + if self.post_smooth: + # Residual around the zero-inited smoother: at init this is the + # identity; during training the 3×3 learns a correction. + x = x + self.smooth(x) + return self.norm(x) + + +def _build_depatch_fourier_basis(r: int, num_freqs: int) -> torch.Tensor: + coords = (torch.arange(r, dtype=torch.float32) + 0.5) / float(r) - 0.5 + yy, xx = torch.meshgrid(coords, coords, indexing='ij') + dirs = ((1.0, 0.0), (0.0, 1.0), (1.0, 1.0), (1.0, -1.0)) + freq_vecs = [] + f = 1 + while len(freq_vecs) < num_freqs: + for dx, dy in dirs: + freq_vecs.append((f * dx, f * dy)) + if len(freq_vecs) >= num_freqs: + break + f += 1 + + terms = [] + for fx, fy in freq_vecs: + phase = 2.0 * math.pi * (fx * xx + fy * yy) + terms.append(torch.sin(phase)) + terms.append(torch.cos(phase)) + basis = torch.stack(terms, dim=0) + rms = basis.square().mean(dim=(1, 2), keepdim=True).sqrt().clamp_min(1e-6) + basis = basis / rms + return basis / math.sqrt(float(basis.shape[0])) + + +class FeatureFourierDepatchResidual(nn.Module): + """Feature-conditioned subpixel Fourier residual for depatch heads.""" + + def __init__(self, in_dim: int, out_channels: int, r: int, + num_freqs: int = 16, kernel_size: int = 3, + bias: bool = True): + super().__init__() + num_freqs = int(num_freqs) + kernel_size = int(kernel_size) + if num_freqs <= 0: + raise ValueError("num_freqs must be positive") + if kernel_size < 1 or kernel_size % 2 != 1: + raise ValueError( + f"kernel_size must be a positive odd integer; got {kernel_size}") + self.r = int(r) + self.out_channels = int(out_channels) + self.num_terms = 2 * num_freqs + self.pad = kernel_size // 2 + self.coeff = nn.Conv2d( + in_dim, out_channels * self.num_terms, + kernel_size=kernel_size, padding=0, bias=bias, + ) + self.reset_to_zero() + self.register_buffer( + 'basis', + _build_depatch_fourier_basis(self.r, num_freqs), + persistent=False, + ) + + def reset_to_zero(self) -> None: + nn.init.zeros_(self.coeff.weight) + if self.coeff.bias is not None: + nn.init.zeros_(self.coeff.bias) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + B, _, H, W = x.shape + if self.pad: + mode = 'reflect' if H > self.pad and W > self.pad else 'replicate' + x = F.pad(x, (self.pad, self.pad, self.pad, self.pad), mode=mode) + coeff = self.coeff(x) + coeff = coeff.view(B, self.out_channels, self.num_terms, H, W) + basis = self.basis.to(device=coeff.device, dtype=coeff.dtype) + y = torch.einsum('bonhw,nuv->bohuwv', coeff, basis) + return y.contiguous().view(B, self.out_channels, H * self.r, W * self.r) + + +class Depatchify(nn.Module): + """Depatchify from /r grid to /1. + + Parameterized by `overlap_mult` (receptive-field multiplier): + • 1 (default): Linear(d → C·r²) + PixelShuffle(r). Each output pixel + is a function of ONE input token. + • 2: ConvTranspose2d(d, C, kernel=2·r, stride=r, padding=r/2). Each + input token paints a 2r×2r patch; output pixel is a sum of 2×2 + neighboring tokens' contributions. + • 3: ConvTranspose2d(d, C, kernel=3·r, stride=r, padding=r). Each + output pixel sums contributions from the 3×3 token neighborhood. + + `overlap=True` is kept as an alias for overlap_mult=3. + + Used for: + • stage-1 aux head: d_baton=512 → 4 channels at /2 (r=8). + • stage-2 final out: d=1280 → 4 channels at /1 (r=16). + + `zero_init_all` zeros the weight + bias — paired with an identity RGB + skip at the call site so at t=0 the module outputs 0. + """ + + def __init__(self, in_dim: int, out_channels: int, r: int, + zero_init_all: bool = True, bias: bool = True, + overlap: bool = False, + overlap_mult: int = 1, + fourier_features: int = 0, + fourier_kernel: int = 3): + super().__init__() + self.r = r + self.out_channels = out_channels + self.in_dim = in_dim + if overlap and overlap_mult == 1: + overlap_mult = 3 # legacy alias + assert overlap_mult >= 1, f"overlap_mult must be >= 1, got {overlap_mult}" + self.overlap_mult = overlap_mult + + if overlap_mult == 1: + self.linear = nn.Linear(in_dim, out_channels * r * r, bias=bias) + if zero_init_all: + nn.init.zeros_(self.linear.weight) + if self.linear.bias is not None: + nn.init.zeros_(self.linear.bias) + self.deconv = None + else: + kernel = overlap_mult * r + padding = (kernel - r) // 2 + assert (kernel - r) % 2 == 0, ( + f"overlap_mult·r - r must be even; kernel={kernel}, r={r}") + self.deconv = nn.ConvTranspose2d(in_dim, out_channels, + kernel_size=kernel, stride=r, + padding=padding, bias=bias) + if zero_init_all: + nn.init.zeros_(self.deconv.weight) + if self.deconv.bias is not None: + nn.init.zeros_(self.deconv.bias) + else: + # Central-kaiming / spill-zero: same init pattern as + # PatchifyConv's overlap case. The r×r central region of + # each kernel maps to the input token's own output block; + # the rest spills into neighboring blocks. Zero the spill + # at init so at step 0 each token only contributes to its + # own block (matches non-overlap baseline), and the + # cross-block blending learns in. + with torch.no_grad(): + c_lo = padding + c_hi = padding + r + mask = torch.zeros(kernel, kernel, + dtype=self.deconv.weight.dtype, + device=self.deconv.weight.device) + mask[c_lo:c_hi, c_lo:c_hi] = 1.0 + self.deconv.weight.data.mul_(mask) + self.linear = None + self.fourier = ( + FeatureFourierDepatchResidual( + in_dim, out_channels, r, + num_freqs=fourier_features, + kernel_size=fourier_kernel, + bias=bias, + ) + if int(fourier_features) > 0 else None + ) + def forward(self, x_tokens_2d: torch.Tensor) -> torch.Tensor: + """x_tokens_2d: (B, C=in_dim, H, W). Returns (B, out_channels, H·r, W·r).""" + if self.overlap_mult > 1: + out = self.deconv(x_tokens_2d) + else: + B, C, H, W = x_tokens_2d.shape + # (B, C, H, W) → (B, H, W, C) → Linear → (B, H, W, C_out·r²) + y = x_tokens_2d.permute(0, 2, 3, 1) + y = self.linear(y) + y = y.permute(0, 3, 1, 2).contiguous() # (B, C_out·r², H, W) + out = F.pixel_shuffle(y, self.r) # (B, C_out, H·r, W·r) + if self.fourier is not None: + out = out + self.fourier(x_tokens_2d) + return out diff --git a/corridorkey/color_utils.py b/corridorkey/color_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..1e9b2ebc9ccb28a9bb6815b8f5a1c29b41e918f9 --- /dev/null +++ b/corridorkey/color_utils.py @@ -0,0 +1,208 @@ +import torch +import numpy as np + +HDR_ASINH_KNEE = 0.008 +HDR_ASINH_REF = 16.0 + +def _is_tensor(x): + return isinstance(x, torch.Tensor) + +def _signed_pow(x, gamma): + """Sign-preserving power. For x ≥ 0 this is identical to `x.pow(gamma)`. + For x < 0 it returns -|x|^gamma — keeps the operation well-defined (no + NaN) and lets gradient flow through negative inputs. + + For the physically-valid [0, 1] inputs produced by the dataset the + behavior is bit-for-bit identical to the previous clamp-based form. + `to_srgb` uses its own epsilon-shifted branch because gamma < 1 has an + infinite mathematical slope at zero. + """ + if _is_tensor(x): + pos = x.clamp(min=0.0).pow(gamma) + neg = -((-x).clamp(min=0.0).pow(gamma)) + return torch.where(x >= 0, pos, neg) + if isinstance(x, np.ndarray): + t = torch.from_numpy(x) + pos = t.clamp(min=0.0).pow(gamma) + neg = -((-t).clamp(min=0.0).pow(gamma)) + return torch.where(t >= 0, pos, neg).numpy() + return np.sign(x) * np.power(np.abs(x), gamma) + + +def linear_to_srgb(x): + """Scene-linear RGB → sRGB (IEC 61966-2-1). Numpy or torch input. + + Uses standard IEC 61966-2-1 equations: + - x <= 0.0031308: 12.92 * x + - x > 0.0031308: 1.055 * x^(1/2.4) - 0.055 + + To prevent gradient NaN at zero during backpropagation in PyTorch, + we clamp the power branch input to a minimum positive threshold. + """ + if _is_tensor(x): + pos_x = x.clamp(min=0.0) + pos_pow = pos_x.clamp(min=0.0031308).pow(1.0 / 2.4) + pos = torch.where(pos_x <= 0.0031308, pos_x * 12.92, 1.055 * pos_pow - 0.055) + + neg_x = (-x).clamp(min=0.0) + neg_pow = neg_x.clamp(min=0.0031308).pow(1.0 / 2.4) + neg = -torch.where(neg_x <= 0.0031308, neg_x * 12.92, 1.055 * neg_pow - 0.055) + return torch.where(x >= 0, pos, neg) + + t = torch.from_numpy(x) + pos_t = t.clamp(min=0.0) + pos_pow = pos_t.clamp(min=0.0031308).pow(1.0 / 2.4) + pos = torch.where(pos_t <= 0.0031308, pos_t * 12.92, 1.055 * pos_pow - 0.055) + + neg_t = (-t).clamp(min=0.0) + neg_pow = neg_t.clamp(min=0.0031308).pow(1.0 / 2.4) + neg = -torch.where(neg_t <= 0.0031308, neg_t * 12.92, 1.055 * neg_pow - 0.055) + t = torch.where(t >= 0, pos, neg) + return t.numpy() + + +def srgb_to_linear(x): + """sRGB → Scene-linear RGB (IEC 61966-2-1). Numpy or torch input. + + Uses standard IEC 61966-2-1 equations: + - x <= 0.04045: x / 12.92 + - x > 0.04045: ((x + 0.055) / 1.055)^2.4 + """ + if _is_tensor(x): + pos_x = x.clamp(min=0.0) + pos_pow = ((pos_x + 0.055) / 1.055).pow(2.4) + pos = torch.where(pos_x <= 0.04045, pos_x / 12.92, pos_pow) + + neg_x = (-x).clamp(min=0.0) + neg_pow = ((neg_x + 0.055) / 1.055).pow(2.4) + neg = -torch.where(neg_x <= 0.04045, neg_x / 12.92, neg_pow) + return torch.where(x >= 0, pos, neg) + + t = torch.from_numpy(x) + pos_t = t.clamp(min=0.0) + pos_pow = ((pos_t + 0.055) / 1.055).pow(2.4) + pos = torch.where(pos_t <= 0.04045, pos_t / 12.92, pos_pow) + + neg_t = (-t).clamp(min=0.0) + neg_pow = ((neg_t + 0.055) / 1.055).pow(2.4) + neg = -torch.where(neg_t <= 0.04045, neg_t / 12.92, neg_pow) + t = torch.where(t >= 0, pos, neg) + return t.numpy() + + +# Backwards compatibility aliases +to_srgb = linear_to_srgb +to_linear = srgb_to_linear + + +def linear_to_asinh(x, knee=HDR_ASINH_KNEE, ref=HDR_ASINH_REF): + """Signed asinh encoding for scene-linear RGB. + + `ref` is a normalization reference, not a clamp. Values above `ref` + encode above 1.0 and remain reversible. This is the model RGB contract for + HDR-capable training: physical compositing stays in decoded linear light, + while the network sees a compressed but exact signed representation. + + Native asinh is already odd and has finite nonzero derivative at zero, so + this intentionally avoids sign/abs branch construction. + """ + knee = float(knee) + ref = float(ref) + denom = np.arcsinh(ref / knee) + if _is_tensor(x): + scale = x.new_tensor(denom) + k = x.new_tensor(knee) + return torch.asinh(x / k) / scale + return np.arcsinh(x / knee) / denom + + +def asinh_to_linear(y, knee=HDR_ASINH_KNEE, ref=HDR_ASINH_REF): + """Inverse of `linear_to_asinh`.""" + knee = float(knee) + ref = float(ref) + denom = np.arcsinh(ref / knee) + if _is_tensor(y): + scale = y.new_tensor(denom) + k = y.new_tensor(knee) + return torch.sinh(y * scale) * k + return np.sinh(y * denom) * knee + + +def linear_to_vfm_sdr(x): + """Tone-limited proxy for foundation models that expect SDR RGB [0, 1].""" + return np.clip(linear_to_srgb(np.maximum(x, 0.0)), 0.0, 1.0) + + +def linear_to_vfm_sdr_torch(x): + """Torch version of `linear_to_vfm_sdr`.""" + return linear_to_srgb(x.clamp(min=0.0)).clamp(0.0, 1.0) + +def premultiply(fg, alpha): + """ + Premultiplies foreground by alpha. + fg: Color [..., C] or [C, ...] + alpha: Alpha [..., 1] or [1, ...] + """ + return fg * alpha + +def unpremultiply(fg, alpha, eps=1e-6): + """ + Un-premultiplies foreground by alpha. + Ref: fg_straight = fg_premul / (alpha + eps) + + Uses max(alpha, 0) as the denominator floor so slightly-negative alpha + (e.g. from Lanczos ringing) can't land on alpha == -eps → denom == 0. + """ + if _is_tensor(fg): + return fg / (alpha.clamp(min=0.0) + eps) + return fg / (np.maximum(alpha, 0.0) + eps) + +def composite_straight(fg, bg, alpha): + """ + Composites Straight FG over BG. + Formula: FG * Alpha + BG * (1 - Alpha) + """ + return fg * alpha + bg * (1.0 - alpha) + +def composite_premul(fg, bg, alpha): + """ + Composites Premultiplied FG over BG. + Formula: FG + BG * (1 - Alpha) + """ + return fg + bg * (1.0 - alpha) + +def rgb_to_yuv(image): + """ + Converts RGB to YUV (Rec. 601). + Input: [..., 3, H, W] or [..., 3] depending on layout. + Supports standard PyTorch BCHW. + """ + if not _is_tensor(image): + raise TypeError("rgb_to_yuv only supports dict/tensor inputs currently") + + # Weights for RGB -> Y + # Rec. 601: 0.299, 0.587, 0.114 + + # Assume BCHW layout if 4 dims + if image.dim() == 4: + r = image[:, 0:1, :, :] + g = image[:, 1:2, :, :] + b = image[:, 2:3, :, :] + elif image.dim() == 3 and image.shape[0] == 3: # CHW + r = image[0:1, :, :] + g = image[1:2, :, :] + b = image[2:3, :, :] + else: + # Last dim conversion + r = image[..., 0] + g = image[..., 1] + b = image[..., 2] + + y = 0.299 * r + 0.587 * g + 0.114 * b + u = 0.492 * (b - y) + v = 0.877 * (r - y) + + if image.dim() >= 3 and image.shape[-3] == 3: # Concatenate along Channel dim + return torch.cat([y, u, v], dim=-3) + else: + return torch.stack([y, u, v], dim=-1) diff --git a/corridorkey/foundations.py b/corridorkey/foundations.py new file mode 100644 index 0000000000000000000000000000000000000000..6cc34e2da57ac42132163db9a6bc65bde354ded3 --- /dev/null +++ b/corridorkey/foundations.py @@ -0,0 +1,327 @@ +"""Frozen foundation-model wrappers for CorridorKey v2. + +Produces per-token features at /64 of the input resolution: + • C-RADIOv4-SO400M: input (H/4, W/4), patch 16 → /64 grid. + • MoGe-ViT-B: input (7H/32, 7W/32), patch 14 → /64 grid. + +Outputs are normalized per-channel via precomputed mean/std (loaded via +`load_stats`); when stats aren't loaded, the normalization is pass-through +(mean=0, std=1). + +Stats are precomputed and shipped alongside the model weights. +""" + +import importlib.util +import os +from pathlib import Path +import sys +import types + +import torch +import torch.nn as nn + + +# --------------------------------------------------------------------------- +# MoGe loader +# --------------------------------------------------------------------------- + + +def build_moge_vitb(device='cuda', checkpoint='Ruicheng/moge-2-vitb-normal'): + from moge.model.v2 import MoGeModel + m = MoGeModel.from_pretrained(checkpoint) + m = m.to(device).eval() + try: + m.enable_pytorch_native_sdpa() + except Exception: + pass + for p in m.parameters(): + p.requires_grad_(False) + return m + + +# --------------------------------------------------------------------------- +# C-RADIO loader +# --------------------------------------------------------------------------- + + +def _load_local_cradio_so400m(repo_path: Path): + """Load bundled C-RADIO remote-code files without HF dynamic-module cache. + + The exported runtime ships a full local HuggingFace snapshot. Loading the + class directly avoids any dependency on the user's global + ~/.cache/huggingface/modules state while still using the official C-RADIO + implementation and safetensors weights from the snapshot. + """ + hf_model_path = repo_path / "hf_model.py" + if not hf_model_path.is_file(): + raise FileNotFoundError(f"C-RADIO snapshot is missing {hf_model_path}") + + package_name = "_corridorkey_cradio_snapshot" + package = sys.modules.get(package_name) + if package is None: + package = types.ModuleType(package_name) + package.__file__ = str(repo_path / "__init__.py") + package.__path__ = [str(repo_path)] + sys.modules[package_name] = package + + module_name = f"{package_name}.hf_model" + module = sys.modules.get(module_name) + if module is None: + spec = importlib.util.spec_from_file_location(module_name, hf_model_path) + if spec is None or spec.loader is None: + raise RuntimeError(f"Could not import bundled C-RADIO module at {hf_model_path}") + module = importlib.util.module_from_spec(spec) + sys.modules[module_name] = module + spec.loader.exec_module(module) + + config = module.RADIOConfig.from_pretrained( + str(repo_path), local_files_only=True, + ) + return module.RADIOModel.from_pretrained( + str(repo_path), config=config, local_files_only=True, + ) + + +def build_cradio_so400m(repo: str = 'nvidia/C-RADIOv4-SO400M', device='cuda'): + """Build frozen C-RADIOv4-SO400M. + + C-RADIO's internal input conditioner expects RGB in [0, 1] unless + `make_preprocessor_external()` has been called. We keep the conditioner + internal so callers can feed the same range used by the rest of this + pipeline. + """ + repo_path = Path(repo) + if repo_path.is_dir(): + m = _load_local_cradio_so400m(repo_path) + else: + from transformers import AutoModel + m = AutoModel.from_pretrained( + repo, trust_remote_code=True, local_files_only=True, + ) + for p in m.parameters(): + p.requires_grad_(False) + return m.eval().to(device) + + +# --------------------------------------------------------------------------- +# Unified wrapper +# --------------------------------------------------------------------------- + + +class FrozenFoundations(nn.Module): + """Frozen C-RADIO + MoGe/RVM feature extractor producing /64 features. + + Stats normalization: outputs are normalized channel-wise with + normalized = (x - mean) / std + using precomputed per-channel buffers (`cradio_mean`, `cradio_std`, + `moge_mean`, `moge_std`, `rvm_mean`, `rvm_std`). When stats file hasn't been loaded the + buffers hold (0, 1) so normalization is the identity. + """ + + def __init__(self, device='cuda', + use_moge=True, + use_cradio: bool = False, + cradio_repo: str = 'nvidia/C-RADIOv4-SO400M', + moge_checkpoint: str = 'Ruicheng/moge-2-vitb-normal', + rvm_variant: str = 'small', + rvm_ckpt: str = None, + stats_path: str = None): + super().__init__() + self.use_moge = use_moge + self.use_cradio = use_cradio + self.cradio_repo = cradio_repo if use_cradio else None + # RVM is implicitly enabled by passing --rvm_ckpt. + self.use_rvm = rvm_ckpt is not None and os.path.isfile(str(rvm_ckpt)) + self.rvm_variant = rvm_variant if self.use_rvm else None + + self.moge_checkpoint = str(moge_checkpoint) if use_moge else None + self.moge = ( + build_moge_vitb(device=device, checkpoint=self.moge_checkpoint) + if use_moge else None + ) + self.cradio = build_cradio_so400m( + repo=cradio_repo, device=device, + ) if use_cradio else None + + # RVM (Recurrent Video MAE from google-deepmind/representations4d). + # Stateful across frames; input in [0, 1] (no ImageNet norm). + if self.use_rvm: + from .rvm import RVMVideoSiamMAE, load_rvm_npz + self.rvm = RVMVideoSiamMAE(variant=rvm_variant).to(device) + load_rvm_npz(self.rvm, rvm_ckpt, strict_load=True, verbose=True) + for p in self.rvm.parameters(): + p.requires_grad_(False) + self.rvm.eval() + else: + self.rvm = None + + # Per-channel output stats (buffers for state-dict persistence). + # Shapes: (C,). Default 0/1 → pass-through. + moge_c = 768 if use_moge else 0 + cradio_c = self._infer_cradio_dim() if use_cradio else 0 + rvm_c = self.rvm.embed_dim if (self.use_rvm and self.rvm is not None) else 0 + self.register_buffer('moge_mean', torch.zeros(max(moge_c, 1))) + self.register_buffer('moge_std', torch.ones(max(moge_c, 1))) + self.register_buffer('rvm_mean', torch.zeros(max(rvm_c, 1))) + self.register_buffer('rvm_std', torch.ones(max(rvm_c, 1))) + if use_cradio: + self.register_buffer('cradio_mean', torch.zeros(cradio_c)) + self.register_buffer('cradio_std', torch.ones(cradio_c)) + + if stats_path is not None and os.path.isfile(stats_path): + self.load_stats(stats_path) + + # Make sure the wrapper's own buffers land on the right device. + self.to(device) + + def train(self, mode: bool = True): + """Keep frozen feature extractors in eval mode at all times.""" + super().train(mode) # sets self.training = mode + if self.moge is not None: + self.moge.eval() + for p in self.moge.parameters(): + p.requires_grad_(False) + if self.rvm is not None: + self.rvm.eval() + for p in self.rvm.parameters(): + p.requires_grad_(False) + if self.cradio is not None: + self.cradio.eval() + for p in self.cradio.parameters(): + p.requires_grad_(False) + return self + + def load_stats(self, path: str): + """Load per-channel mean/std from a .safetensors (or legacy .pt) file. + File format: + {'cradio_mean': (C_c,), 'cradio_std': (C_c,), + 'moge_mean': (C_m,), 'moge_std': (C_m,), + 'rvm_mean': (C_r,), 'rvm_std': (C_r,)} + Silently skips any key we don't have a buffer for.""" + if str(path).endswith('.safetensors'): + from safetensors.torch import load_file + stats = load_file(str(path), device='cpu') + else: + stats = torch.load(path, map_location='cpu', weights_only=False) + for key in ('moge_mean', 'moge_std', 'rvm_mean', 'rvm_std', + 'cradio_mean', 'cradio_std'): + if key in stats and hasattr(self, key): + buf = getattr(self, key) + val = stats[key].to(buf.device, buf.dtype) + if val.shape != buf.shape: + print(f"[FrozenFoundations] {key}: stats shape " + f"{tuple(val.shape)} != buffer shape {tuple(buf.shape)} — skipping") + continue + buf.copy_(val) + print(f"[FrozenFoundations] loaded stats from {path}") + + @torch.no_grad() + def forward(self, rgb_moge=None, rgb_cradio=None, rgb_rvm=None, + rvm_state=None): + """C-RADIO and MoGe consume (B·T, 3, ...) 4D tensors (per-frame IID). + RVM consumes (B, T, 3, H_rvm, W_rvm) 5D because it is recurrent across T. + + Args: + rgb_cradio: (B·T, 3, H/4, W/4) — if C-RADIO enabled. + rgb_moge: (B·T, 3, 7H/32, 7W/32) — if MoGe enabled. + rgb_rvm: (B, T, 3, H_rvm, W_rvm) — if RVM enabled. Typically + the same H/4 input used by C-RADIO. + rvm_state: optional (B, 1+N_rvm, D_rvm) carry state for RVM. + At training time, pass None (fresh per batch). + + Returns: + Dict with channel-normalized features: + 'moge': (B·T, C_m, H/64, W/64) + 'cradio': (B·T, C_c, H/64, W/64) + 'rvm': (B·T, C_r, h_rvm, w_rvm) (grid at RVM input / 16) + 'rvm_state': (B, 1+N_rvm, D_rvm) final state, for streaming + inference across windows (ignored in training). + """ + out = {} + + if self.cradio is not None: + assert rgb_cradio is not None, "C-RADIO enabled but rgb_cradio not provided" + try: + cradio_out = self.cradio(rgb_cradio, feature_fmt='NCHW') + except TypeError: + cradio_out = self.cradio(rgb_cradio) + cradio_feat = cradio_out.features + if cradio_feat.dim() == 3: + Bc, n_tok, Cc = cradio_feat.shape + patch = int(getattr(self.cradio, 'patch_size', 16)) + hc, wc = rgb_cradio.shape[-2] // patch, rgb_cradio.shape[-1] // patch + if n_tok != hc * wc: + raise RuntimeError( + f"C-RADIO token count {n_tok} != expected grid {hc}x{wc}") + cradio_feat = cradio_feat.transpose(1, 2).reshape(Bc, Cc, hc, wc) + cradio_feat = cradio_feat.float() + mean = self.cradio_mean.to( + cradio_feat.device, cradio_feat.dtype, + ).view(1, -1, 1, 1) + std = self.cradio_std.to( + cradio_feat.device, cradio_feat.dtype, + ).view(1, -1, 1, 1) + cradio_feat = (cradio_feat - mean) / std.clamp(min=1e-6) + out['cradio'] = cradio_feat + + if self.moge is not None: + assert rgb_moge is not None, "MoGe enabled but rgb_moge not provided" + Bm, _, Hm, Wm = rgb_moge.shape + base_h, base_w = Hm // 14, Wm // 14 + moge_feat, _ = self.moge.encoder( + rgb_moge, base_h, base_w, return_class_token=True, + ) + moge_feat = moge_feat.float() + mean = self.moge_mean.to(moge_feat.device, moge_feat.dtype).view(1, -1, 1, 1) + std = self.moge_std.to(moge_feat.device, moge_feat.dtype).view(1, -1, 1, 1) + moge_feat = (moge_feat - mean) / std.clamp(min=1e-6) + out['moge'] = moge_feat + + # RVM is allowed to be skipped when rgb_rvm is None even if the + # model is enabled — useful when the caller pre-computes RVM + # features externally (e.g. inference's full-clip stateful pass) + # and supplies them to stage1 directly. In that case foundations + # shouldn't also run RVM inline. + if self.rvm is not None and rgb_rvm is not None: + # rgb_rvm: (B, T, 3, H, W). No mean/std norm — RVM wants [0, 1]. + rvm_feat, new_state = self.rvm(rgb_rvm, state=rvm_state) # (B, T, D, h, w) + B_r, T_r, D_r, h_r, w_r = rvm_feat.shape + rvm_feat = rvm_feat.reshape(B_r * T_r, D_r, h_r, w_r).float() + mean = self.rvm_mean.to(rvm_feat.device, rvm_feat.dtype).view(1, -1, 1, 1) + std = self.rvm_std.to(rvm_feat.device, rvm_feat.dtype).view(1, -1, 1, 1) + rvm_feat = (rvm_feat - mean) / std.clamp(min=1e-6) + out['rvm'] = rvm_feat + out['rvm_state'] = new_state + + return out + + def _infer_cradio_dim(self): + if self.cradio is None: + return 0 + for obj in (self.cradio, getattr(self.cradio, 'radio_model', None), + getattr(self.cradio, 'model', None)): + dim = getattr(obj, 'embed_dim', None) if obj is not None else None + if dim is not None: + return int(dim) + raise AttributeError("Unable to infer C-RADIO embed_dim") + + @property + def moge_dim(self): + return 768 if self.use_moge else 0 + + @property + def cradio_dim(self): + if not self.use_cradio: + return 0 + return self._infer_cradio_dim() + + @property + def rvm_dim(self): + if not self.use_rvm or self.rvm is None: + return 0 + return self.rvm.embed_dim + + @property + def vfm_dim(self): + """Concatenated foundation channel count.""" + return self.cradio_dim + self.moge_dim + self.rvm_dim diff --git a/corridorkey/model.py b/corridorkey/model.py new file mode 100644 index 0000000000000000000000000000000000000000..f48c7dfe42c6ac04d0abf8b4be125a7fdcb01c67 --- /dev/null +++ b/corridorkey/model.py @@ -0,0 +1,307 @@ +"""Top-level model assembly for CorridorKey v2. + +Forward contract: + rgb: (B, T, 3, H, W) signed asinh encoded scene-linear RGB + hint: (B, T, 1, H, W) in [0, 1] + rgb_cradio:(B, T, 3, H/4, W/4) for C-RADIO/RVM (patch 16 → /64 grid) + rgb_moge: (B, T, 3, 7H/32, 7W/32) for MoGe (patch 14 → /64 grid) + t_stride: float — temporal RoPE scaling factor + +H and W must be multiples of 64 (shape_quantum=64). The dataset +guarantees this. + +Output dict: + 'alpha': (B, T, 1, H, W) — stage-2 refined + 'fg': (B, T, 3, H, W) asinh encoded — stage-2 refined + 'alpha_aux': (B, T, 1, H/2, W/2) — stage-1 auxiliary prediction + 'fg_aux': (B, T, 3, H/2, W/2) — stage-1 auxiliary prediction + 'alpha_base': optional (B, T, 1, H, W) — pre-/4-refine stage-2 output + 'fg_base': optional (B, T, 3, H, W) — pre-/4-refine stage-2 output + 'alpha_bottleneck': optional (B, T, 1, H/8, W/8) — /64 supervision head + 'fg_bottleneck': optional (B, T, 3, H/8, W/8) — /64 supervision head + 'baton': (B, T, d_baton, H/16, W/16) — debug/intermediate +""" + +import torch +import torch.nn as nn + +from .foundations import FrozenFoundations +from .stage1 import Stage1 +from .stage2 import Stage2 +from .rope import refresh_all_augs + + +class CorridorKeyV3(nn.Module): + """Transformer-native two-stage video matting model. + + Stage 1 produces a /16 baton (d=512) plus an auxiliary /2 prediction + for deep supervision. Stage 2 refines the baton + raw RGB to a /1 + alpha+FG output via pure-transformer Swin-2D + depatchify. + + Temporal mixing lives primarily in stage 1: full spatiotemporal attention + at /64, Swin-3D at /16, and optional /64 recurrent memory sites for + streaming identity/decision carry. + """ + + def __init__( + self, + # Stage 1 + d_16: int = 512, + d_64: int = 1536, + n_enc_16: int = 2, + n_body_64: int = 10, + n_dec_16: int = 6, + num_heads_16: int = 8, + num_heads_64: int = 16, + window_HW: int = 16, + stage1_local_mix_kernel: int = 7, + stage1_local_mix_init_std: float | None = None, + stage1_body64_local_mix_kernel: int = 7, + stage1_body64_local_mix_init_std: float | None = None, + stage1_body64_mlp_hidden: int | None = None, + stage1_body64_direct_concat: bool = False, + stage1_recurrent_64_sites: tuple[int, ...] | None = None, + stage1_hint_adaln_blocks: int = 2, + stage1_mid_16_blocks: int = 4, + # Stage 2 + d_stage2: int = 1280, + n_blocks_stage2: int = 10, + n_final_blocks_stage2: int = 2, + num_heads_stage2: int = 16, + window_size_stage2: int = 16, + stage2_local_mix_kernel: int = 7, + stage2_local_mix_init_std: float | None = None, + refine4_blocks: int = 0, + refine4_width: int = 96, + refine4_token_channels: int = 48, + refine4_kernel: int = 7, + refine4_expansion: float = 2.0, + # Shared + mlp_mult: float = 2.5, + mlp_dw_kernel: int = 0, + qkv_dw_kernel: int = 7, + qkv_dw_fraction: float = 0.25, + # Foundations + use_moge: bool = True, + use_cradio: bool = False, + cradio_repo: str = 'nvidia/C-RADIOv4-SO400M', + moge_checkpoint: str = 'Ruicheng/moge-2-vitb-normal', + # RVM is enabled by passing a ckpt path. RVM has patch 16, so + # we feed it the same H/4-sized input C-RADIO sees; features land at + # the /64 grid naturally, no resize or aspect-distortion needed. + rvm_variant: str = 'small', + rvm_ckpt: str = None, + foundation_stats_path: str = None, + device: str = 'cuda', + # Training + gradient_checkpoint: bool = True, + # Architecture ablation knob: overlap patchify/depatchify. + # overlap_patchify is a legacy bool alias (True → mult=3). + # overlap_mult: 1 (no overlap), 2 (half), 3 (full 3×3 neighborhood). + overlap_patchify: bool = False, + overlap_mult: int = 2, + depatch_fourier_features: int = 0, + depatch_fourier_kernel: int = 3, + # MAE-style pretraining hooks plus the /64 bottleneck supervision head. + # The bottleneck head default matches phase-2 training. + enable_mae_mask_tokens: bool = False, + bottleneck_head_stride: int | None = 8, + # DINOv3-style RoPE augmentations (training-time, per-forward coherent) + rope_shift_coords: float = None, + rope_jitter_coords: float = None, + rope_rescale_coords: float = None, + rope_base: float = 100.0, + ): + super().__init__() + self.gradient_checkpoint = gradient_checkpoint + self.d_16 = d_16 + + self.foundations = FrozenFoundations( + device=device, + use_moge=use_moge, + use_cradio=use_cradio, cradio_repo=cradio_repo, + moge_checkpoint=moge_checkpoint, + rvm_variant=rvm_variant, + rvm_ckpt=rvm_ckpt, + stats_path=foundation_stats_path, + ) + + self.stage1 = Stage1( + d_16=d_16, d_64=d_64, + cradio_dim=self.foundations.cradio_dim, + moge_dim=self.foundations.moge_dim, + rvm_dim=self.foundations.rvm_dim, + n_enc_16=n_enc_16, n_body_64=n_body_64, n_dec_16=n_dec_16, + num_heads_16=num_heads_16, num_heads_64=num_heads_64, + window_HW=window_HW, mlp_mult=mlp_mult, + local_mix_kernel=stage1_local_mix_kernel, + local_mix_init_std=stage1_local_mix_init_std, + body64_local_mix_kernel=stage1_body64_local_mix_kernel, + body64_local_mix_init_std=stage1_body64_local_mix_init_std, + body64_mlp_hidden=stage1_body64_mlp_hidden, + body64_direct_concat=stage1_body64_direct_concat, + recurrent_64_sites=stage1_recurrent_64_sites, + hint_adaln_blocks=stage1_hint_adaln_blocks, + mid_16_blocks=stage1_mid_16_blocks, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=qkv_dw_kernel, + qkv_dw_fraction=qkv_dw_fraction, + gradient_checkpoint=gradient_checkpoint, + overlap_patchify=overlap_patchify, + overlap_mult=overlap_mult, + depatch_fourier_features=depatch_fourier_features, + depatch_fourier_kernel=depatch_fourier_kernel, + enable_mae_mask_tokens=enable_mae_mask_tokens, + bottleneck_head_stride=bottleneck_head_stride, + rope_shift_coords=rope_shift_coords, + rope_jitter_coords=rope_jitter_coords, + rope_rescale_coords=rope_rescale_coords, + rope_base=rope_base, + ) + + self.stage2 = Stage2( + d=d_stage2, d_baton=d_16, + num_heads=num_heads_stage2, + num_blocks=n_blocks_stage2, + num_final_blocks=n_final_blocks_stage2, + window_size=window_size_stage2, + mlp_mult=mlp_mult, + gradient_checkpoint=gradient_checkpoint, + overlap_patchify=overlap_patchify, + overlap_mult=overlap_mult, + depatch_fourier_features=depatch_fourier_features, + depatch_fourier_kernel=depatch_fourier_kernel, + enable_mae_mask_tokens=enable_mae_mask_tokens, + local_mix_kernel=stage2_local_mix_kernel, + local_mix_init_std=stage2_local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=qkv_dw_kernel, + qkv_dw_fraction=qkv_dw_fraction, + refine4_blocks=refine4_blocks, + refine4_width=refine4_width, + refine4_token_channels=refine4_token_channels, + refine4_kernel=refine4_kernel, + refine4_expansion=refine4_expansion, + rope_shift_coords=rope_shift_coords, + rope_jitter_coords=rope_jitter_coords, + rope_rescale_coords=rope_rescale_coords, + rope_base=rope_base, + ) + + def forward( + self, + rgb: torch.Tensor, + hint: torch.Tensor, + rgb_cradio: 'torch.Tensor | None', + rgb_moge: torch.Tensor, + rgb_rvm: 'torch.Tensor | None' = None, + rvm_state: 'torch.Tensor | None' = None, + t_stride: float = 1.0, + mae_mask_16: 'torch.Tensor | None' = None, + mae_mask_64: 'torch.Tensor | None' = None, + mae_vfm_mask_64: 'torch.Tensor | None' = None, + hint_quality: 'torch.Tensor | None' = None, + stage1_recurrent_states: 'tuple[torch.Tensor | None, ...] | None' = None, + ) -> dict: + B, T, _, H, W = rgb.shape + assert H % 64 == 0 and W % 64 == 0, ( + f"H, W must be multiples of 64 (shape_quantum); got {(H, W)}.") + if (mae_mask_16 is None) != (mae_mask_64 is None): + raise ValueError("mae_mask_16 and mae_mask_64 must be passed together") + if mae_vfm_mask_64 is not None and mae_mask_64 is None: + raise ValueError("mae_vfm_mask_64 requires mae_mask_64") + + # Refresh RoPE augmentations ONCE per forward so every RoPE call + # inside stage1/stage2 (across all blocks) uses the same sample. + # In eval mode this clears the state to identity (deterministic). + refresh_all_augs(self, device=rgb.device) + + # Foundations — per-frame, no gradients. + BT = B * T + rgb_moge_flat = rgb_moge.reshape(BT, 3, *rgb_moge.shape[-2:]) + rgb_cradio_flat = ( + rgb_cradio.reshape(BT, 3, *rgb_cradio.shape[-2:]) + if rgb_cradio is not None else None + ) + # RVM input: feed the same H/4-sized tensor C-RADIO sees. RVM has + # patch 16, so the patch grid lands on /64 directly — + # no resize, no aspect distortion. If caller supplied a custom + # rgb_rvm explicitly, honor that instead. + if self.foundations.rvm is not None and rgb_rvm is None: + assert rgb_cradio is not None, ( + "RVM inline extraction requires an H/4 C-RADIO/RVM input") + H_e, W_e = rgb_cradio.shape[-2:] + rgb_rvm = rgb_cradio.reshape(B, T, 3, H_e, W_e) + with torch.no_grad(): + foundation_feats = self.foundations( + rgb_moge=rgb_moge_flat, + rgb_cradio=rgb_cradio_flat, + rgb_rvm=rgb_rvm, rvm_state=rvm_state, + ) + cradio_feat = foundation_feats.get('cradio') + moge_feat = foundation_feats.get('moge') + rvm_feat = foundation_feats.get('rvm') + rvm_state_out = foundation_feats.get('rvm_state') + if cradio_feat is not None: + cradio_feat = cradio_feat.reshape(B, T, -1, H // 64, W // 64) + else: + cradio_feat = rgb.new_zeros(B, T, 0, H // 64, W // 64) + if moge_feat is not None: + moge_feat = moge_feat.reshape(B, T, -1, H // 64, W // 64) + else: + moge_feat = rgb.new_zeros(B, T, 0, H // 64, W // 64) + if rvm_feat is not None: + rvm_feat = rvm_feat.reshape(B, T, rvm_feat.shape[-3], + rvm_feat.shape[-2], rvm_feat.shape[-1]) + + # Stage 1 & 2 — per-block gradient checkpointing is done INSIDE each + # stage's forward (see Stage1._ckpt_block, Stage2.run_block). No outer + # stage-level wrap — per-block peak activations are much smaller than + # stage-level, and double-wrapping would just cause nested recompute. + stage1_out = self.stage1( + rgb, hint, moge_feat, + cradio_feat=cradio_feat, + rvm_feat=rvm_feat, t_stride=t_stride, + mae_mask_16=mae_mask_16, mae_mask_64=mae_mask_64, + mae_vfm_mask_64=mae_vfm_mask_64, + hint_quality=hint_quality, + recurrent_states=stage1_recurrent_states, + ) + baton = stage1_out['baton'] # (B, T, d_16, H/16, W/16) + aux = stage1_out['aux'] # (B, T, 4, H/2, W/2) + bottleneck = stage1_out.get('bottleneck') + recurrent_states_out = stage1_out.get('recurrent_states') + del stage1_out, foundation_feats + del cradio_feat, moge_feat, rvm_feat + del rgb_moge_flat, rgb_cradio_flat, rgb_rvm + + logits_out = self.stage2(rgb, baton, mae_mask_16=mae_mask_16) + if isinstance(logits_out, tuple): + logits, logits_base = logits_out + else: + logits, logits_base = logits_out, None + + # Split outputs + alpha = logits[:, :, 0:1] + fg = logits[:, :, 1:4] + alpha_aux = aux[:, :, 0:1] + fg_aux = aux[:, :, 1:4] + + out = { + 'alpha': alpha, + 'fg': fg, + 'alpha_aux': alpha_aux, + 'fg_aux': fg_aux, + 'baton': baton, + } + if logits_base is not None: + out['alpha_base'] = logits_base[:, :, 0:1] + out['fg_base'] = logits_base[:, :, 1:4] + if bottleneck is not None: + out['alpha_bottleneck'] = bottleneck[:, :, 0:1] + out['fg_bottleneck'] = bottleneck[:, :, 1:4] + if recurrent_states_out is not None: + out['stage1_recurrent_states'] = recurrent_states_out + if rvm_state_out is not None: + out['rvm_state'] = rvm_state_out + return out diff --git a/corridorkey/rope.py b/corridorkey/rope.py new file mode 100644 index 0000000000000000000000000000000000000000..19cf265fdaa87acce02248a19816f8a1e3a5a5eb --- /dev/null +++ b/corridorkey/rope.py @@ -0,0 +1,320 @@ +"""Rotary Position Embeddings — 2D (stage 2) and 3D (stage 1) variants. + +DINOv3/EUPE-style: normalized [-1, 1] cell-center coordinates with +training-time augmentations (shift, per-axis jitter, global rescale). + +CorridorKey v2 uses: + • 3D RoPE (T, H, W) inside full-attention body at /64 and inside Swin-3D + windows at /16 (stage 1). + • 2D RoPE (H, W) inside Swin-2D windows at /16 (stage 2; no temporal). + +Periods (lengths in normalized-coord units): + periods[i] = base ** (2i / (D_axis)) for i in [0, D_axis // 2) + angles = 2π · coord / period +where `coord ∈ [-1, 1]` is the cell-center coord of a token in the +(sub-)grid and `D_axis` is the portion of head_dim assigned to this axis. + +Head-dim split: + 2D: head_dim/4 periods per axis × 2 axes → head_dim/2 cos/sin pairs + 3D: head_dim/2 cos/sin pairs split as evenly as possible across T/H/W +(2D asserts divisibility by 4; 3D only requires an even head_dim) + +Application format (kept from original): + x[0::2] / x[1::2] interleave — cos/sin tensors are (N, head_dim//2), + one value per pair. + +Training augmentations, sampled once per forward pass via refresh_augs() +so every RoPE call during the same forward gets a CONSISTENT positional +encoding: + • shift_coords (s): per-axis additive shift ~ U[-s, s] + • jitter_coords (J): per-axis multiplicative log-uniform in [1/J, J] + (aspect-ratio BREAKING — each axis jitters + independently) + • rescale_coords (R): global multiplicative log-uniform in [1/R, R] + (aspect-ratio PRESERVING — same scale on every + axis) +All disabled at eval. + +Convention: + • Token ordering: (t, h, w) with w varying fastest for 3D; + (h, w) with w fastest for 2D. + • All three axes use the same `base` (default 100). + • Coord normalization per EUPE's "separate" mode: coord_a = (i+0.5)/N_a. +""" + +import math +from typing import List, Optional, Tuple + +import torch +import torch.nn as nn + + +def _compute_periods(dim: int, base: float) -> torch.Tensor: + """Period lengths in normalized-coord units. Returns (dim//2,) tensor.""" + assert dim % 2 == 0, f"_compute_periods dim={dim} must be even" + exps = torch.arange(0, dim // 2, dtype=torch.float32) * 2.0 / dim + return base ** exps # Range: [1, base^((dim-2)/dim)] + + +def _normalized_coords(n: int, device=None, dtype=torch.float32) -> torch.Tensor: + """Cell-center normalized coords in [-1, 1] for n tokens along an axis. + Matches DINOv3 / EUPE 'separate' normalization: (i+0.5)/n then 2x-1.""" + x = (torch.arange(n, device=device, dtype=dtype) + 0.5) / n # [0.5/n, (n-0.5)/n] + return 2.0 * x - 1.0 + + +def _apply_axis_rotary(coord: torch.Tensor, periods: torch.Tensor + ) -> Tuple[torch.Tensor, torch.Tensor]: + """Given coord (N,) in normalized units and periods (D_axis//2,), + return (cos, sin) of shape (N, D_axis//2).""" + angles = 2.0 * math.pi * coord.unsqueeze(1) / periods.unsqueeze(0) + return torch.cos(angles), torch.sin(angles) + + +def apply_rotary(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor + ) -> torch.Tensor: + """Apply rotation to the head dimension of `x`. + + Args: + x: (..., N, head_dim) — queries or keys. + cos/sin: (N, head_dim // 2) — frequency tables. + + Returns rotated tensor of same shape as `x`. + """ + while cos.dim() < x.dim(): + cos = cos.unsqueeze(0) + sin = sin.unsqueeze(0) + cos = cos.to(x.dtype) + sin = sin.to(x.dtype) + x1 = x[..., 0::2] + x2 = x[..., 1::2] + out1 = x1 * cos - x2 * sin + out2 = x1 * sin + x2 * cos + return torch.stack([out1, out2], dim=-1).flatten(-2) + + +# --------------------------------------------------------------------------- +# Augmentation state (shared logic between RoPE2D and RoPE3D) +# --------------------------------------------------------------------------- + + +class _RopeAugBase(nn.Module): + """Holds training-time augmentation params and a per-forward cached + sample. Subclasses fill in axis geometry + coord construction.""" + + def __init__(self, base: float = 100.0, + shift_coords: Optional[float] = None, + jitter_coords: Optional[float] = None, + rescale_coords: Optional[float] = None, + n_axes: int = 2): + super().__init__() + self.base = base + self.shift_coords = shift_coords + self.jitter_coords = jitter_coords + self.rescale_coords = rescale_coords + self.n_axes = n_axes + # Cached per-forward augmentation: (shift_vec, jitter_log_vec, rescale_log_scalar). + # `None` means "use identity" (eval-time or uninitialized). + self._aug_shift: Optional[List[float]] = None + self._aug_jitter_log: Optional[List[float]] = None + self._aug_rescale_log: Optional[float] = None + + def refresh_augs(self, device=None, force_eval: bool = False) -> None: + """Call at the start of each model forward during training; clears + the aug state (identity) during eval.""" + if force_eval or not self.training: + self._aug_shift = None + self._aug_jitter_log = None + self._aug_rescale_log = None + return + if self.shift_coords is not None and self.shift_coords > 0: + s = float(self.shift_coords) + self._aug_shift = (torch.empty(self.n_axes) + .uniform_(-s, s).tolist()) + else: + self._aug_shift = None + if self.jitter_coords is not None and self.jitter_coords > 1.0: + lg = math.log(float(self.jitter_coords)) + self._aug_jitter_log = (torch.empty(self.n_axes) + .uniform_(-lg, lg).tolist()) + else: + self._aug_jitter_log = None + if self.rescale_coords is not None and self.rescale_coords > 1.0: + lg = math.log(float(self.rescale_coords)) + self._aug_rescale_log = float(torch.empty(1) + .uniform_(-lg, lg).item()) + else: + self._aug_rescale_log = None + + def _aug_coord(self, coord: torch.Tensor, axis_idx: int) -> torch.Tensor: + """Apply current-forward augs to a 1D coord tensor for axis `axis_idx`.""" + out = coord + if self._aug_shift is not None: + out = out + self._aug_shift[axis_idx] + if self._aug_jitter_log is not None: + out = out * math.exp(self._aug_jitter_log[axis_idx]) + if self._aug_rescale_log is not None: + out = out * math.exp(self._aug_rescale_log) + return out + + +# --------------------------------------------------------------------------- +# RoPE3D — used by stage 1 (Swin-3D enc/dec at /16, Full-ST body at /64) +# --------------------------------------------------------------------------- + + +class RoPE3D(_RopeAugBase): + """3D Rotary Position Embedding for spatiotemporal tokens. + + head_dim split: rotary pairs are split nearly evenly across T/H/W. + Exact thirds are used when available; otherwise the extra pairs are given + to H/W first, which lets power-of-two head dims such as 64 work cleanly. + """ + + def __init__(self, head_dim: int, + base: float = 100.0, + shift_coords: Optional[float] = None, + jitter_coords: Optional[float] = None, + rescale_coords: Optional[float] = None): + super().__init__(base=base, + shift_coords=shift_coords, + jitter_coords=jitter_coords, + rescale_coords=rescale_coords, + n_axes=3) + assert head_dim % 2 == 0, ( + f"3D RoPE head_dim={head_dim} must be even for cos/sin pairs.") + self.head_dim = head_dim + total_pairs = head_dim // 2 + base_pairs = total_pairs // 3 + rem = total_pairs - base_pairs * 3 + pairs_t = base_pairs + pairs_h = base_pairs + (1 if rem >= 2 else 0) + pairs_w = base_pairs + (1 if rem >= 1 else 0) + self.dim_t = pairs_t * 2 + self.dim_h = pairs_h * 2 + self.dim_w = pairs_w * 2 + + self.register_buffer('periods_t', + _compute_periods(self.dim_t, base), + persistent=False) + self.register_buffer('periods_h', + _compute_periods(self.dim_h, base), + persistent=False) + self.register_buffer('periods_w', + _compute_periods(self.dim_w, base), + persistent=False) + + def get_cos_sin( + self, + num_t: int, num_h: int, num_w: int, + device: torch.device = None, + t_stride: float = 1.0, + # Legacy args kept for call-site compat; ignored in the normalized + # formulation (Swin windows use local-to-window normalization). + t_offset: int = 0, h_offset: int = 0, w_offset: int = 0, + ) -> Tuple[torch.Tensor, torch.Tensor]: + """Build (num_t * num_h * num_w, head_dim // 2) cos and sin tables. + + Positions are cell-center normalized to [-1, 1] per axis, optionally + augmented by the cached refresh_augs() sample. The `t_stride` scales + the temporal axis post-normalization (dataset-level frame-stride / + frame-rate knob, orthogonal to the training jitter). + """ + device = device or self.periods_t.device + coord_t = _normalized_coords(num_t, device=device) * t_stride + coord_h = _normalized_coords(num_h, device=device) + coord_w = _normalized_coords(num_w, device=device) + coord_t = self._aug_coord(coord_t, axis_idx=0) + coord_h = self._aug_coord(coord_h, axis_idx=1) + coord_w = self._aug_coord(coord_w, axis_idx=2) + + cos_t, sin_t = _apply_axis_rotary(coord_t, self.periods_t.to(device)) + cos_h, sin_h = _apply_axis_rotary(coord_h, self.periods_h.to(device)) + cos_w, sin_w = _apply_axis_rotary(coord_w, self.periods_w.to(device)) + + N = num_t * num_h * num_w + cos_t_exp = cos_t[:, None, None, :].expand(num_t, num_h, num_w, -1).reshape(N, -1) + sin_t_exp = sin_t[:, None, None, :].expand(num_t, num_h, num_w, -1).reshape(N, -1) + cos_h_exp = cos_h[None, :, None, :].expand(num_t, num_h, num_w, -1).reshape(N, -1) + sin_h_exp = sin_h[None, :, None, :].expand(num_t, num_h, num_w, -1).reshape(N, -1) + cos_w_exp = cos_w[None, None, :, :].expand(num_t, num_h, num_w, -1).reshape(N, -1) + sin_w_exp = sin_w[None, None, :, :].expand(num_t, num_h, num_w, -1).reshape(N, -1) + + cos = torch.cat([cos_t_exp, cos_h_exp, cos_w_exp], dim=-1) + sin = torch.cat([sin_t_exp, sin_h_exp, sin_w_exp], dim=-1) + return cos, sin + + +# --------------------------------------------------------------------------- +# RoPE2D — used by stage 2 (Swin-2D at /16) +# --------------------------------------------------------------------------- + + +class RoPE2D(_RopeAugBase): + """2D Rotary Position Embedding for spatial-only tokens (stage 2). + + head_dim split evenly between H and W. head_dim must be divisible by 4. + """ + + def __init__(self, head_dim: int, + base: float = 100.0, + shift_coords: Optional[float] = None, + jitter_coords: Optional[float] = None, + rescale_coords: Optional[float] = None): + super().__init__(base=base, + shift_coords=shift_coords, + jitter_coords=jitter_coords, + rescale_coords=rescale_coords, + n_axes=2) + assert head_dim % 4 == 0, ( + f"2D RoPE head_dim={head_dim} must be divisible by 4.") + self.head_dim = head_dim + dim_each = head_dim // 2 + self.dim_h = dim_each + self.dim_w = dim_each + self.register_buffer('periods_h', + _compute_periods(dim_each, base), + persistent=False) + self.register_buffer('periods_w', + _compute_periods(dim_each, base), + persistent=False) + + def get_cos_sin( + self, num_h: int, num_w: int, + device: torch.device = None, + # Legacy kwargs kept for call-site compat; unused in normalized form. + h_offset: int = 0, w_offset: int = 0, + ) -> Tuple[torch.Tensor, torch.Tensor]: + """Build (num_h * num_w, head_dim // 2) cos/sin tables.""" + device = device or self.periods_h.device + coord_h = _normalized_coords(num_h, device=device) + coord_w = _normalized_coords(num_w, device=device) + coord_h = self._aug_coord(coord_h, axis_idx=0) + coord_w = self._aug_coord(coord_w, axis_idx=1) + + cos_h, sin_h = _apply_axis_rotary(coord_h, self.periods_h.to(device)) + cos_w, sin_w = _apply_axis_rotary(coord_w, self.periods_w.to(device)) + N = num_h * num_w + cos_h_exp = cos_h[:, None, :].expand(num_h, num_w, -1).reshape(N, -1) + sin_h_exp = sin_h[:, None, :].expand(num_h, num_w, -1).reshape(N, -1) + cos_w_exp = cos_w[None, :, :].expand(num_h, num_w, -1).reshape(N, -1) + sin_w_exp = sin_w[None, :, :].expand(num_h, num_w, -1).reshape(N, -1) + cos = torch.cat([cos_h_exp, cos_w_exp], dim=-1) + sin = torch.cat([sin_h_exp, sin_w_exp], dim=-1) + return cos, sin + + +# --------------------------------------------------------------------------- +# Convenience: refresh all RoPE modules' augs in one call +# --------------------------------------------------------------------------- + + +def refresh_all_augs(module: nn.Module, device=None, force_eval: bool = False) -> None: + """Walk a module tree and call refresh_augs on every RoPE instance. + + Invoke once at the top of each model forward pass so all RoPE modules + in the model get a COHERENT augmentation sample for this step. + """ + for m in module.modules(): + if isinstance(m, _RopeAugBase): + m.refresh_augs(device=device, force_eval=force_eval) diff --git a/corridorkey/runtime.py b/corridorkey/runtime.py new file mode 100644 index 0000000000000000000000000000000000000000..aa74162ec20cd6b23eeb49042d83772ec9cc33ce --- /dev/null +++ b/corridorkey/runtime.py @@ -0,0 +1,937 @@ +from __future__ import annotations + +import json +import os +import sys +import tempfile +from concurrent.futures import ThreadPoolExecutor +from dataclasses import dataclass, field +from pathlib import Path + +import numpy as np +import torch +import torch.nn.functional as F + +from . import color_utils as cu +from .model import CorridorKeyV3 +from .video_io import ( + ExrSequenceWriter, + ThreadedVideoWriter, + WriterConfig, + apply_linear_adjust, + frame_paths, + has_linear_adjust, + iter_frame_dir_chunks, + iter_video_frame_chunks, + load_initial_hint_frame, + make_checker, + make_hint_reader, + none_path, + probe_frame_dir_meta, + probe_video_meta, + raw_clip_name, + safe_remove, + taper_weights, + window_starts, + SequentialFrameDirWindowReader, + SequentialVideoWindowReader, +) + + +@dataclass +class RuntimePaths: + package_root: Path + weights: Path + foundation_stats: Path + rvm_ckpt: Path + moge_checkpoint: Path + cradio_dir: Path + + @staticmethod + def _resolve_user_or_package_path(root: Path, value, default: str) -> Path: + if value is None: + return (root / default).resolve() + path = Path(value) + if path.is_absolute(): + return path.resolve() + cwd_path = (Path.cwd() / path).resolve() + if cwd_path.exists(): + return cwd_path + return (root / path).resolve() + + @staticmethod + def _prefer_safetensors(root: Path, stem: str) -> str: + # Legacy pickle .pt files are still accepted if that is all that is present. + if not (root / f"{stem}.safetensors").exists() and (root / f"{stem}.pt").exists(): + return f"{stem}.pt" + return f"{stem}.safetensors" + + @classmethod + def from_package(cls, package_root: str | os.PathLike, + weights: str | os.PathLike | None = None): + root = Path(package_root).resolve() + return cls( + package_root=root, + weights=cls._resolve_user_or_package_path( + root, weights, cls._prefer_safetensors(root, "weights/corridorkey_v2")), + foundation_stats=( + root / cls._prefer_safetensors(root, "weights/foundation_stats")).resolve(), + rvm_ckpt=(root / "vfms/rvm/pretrain_rvm_small16_256_204031069.npz").resolve(), + moge_checkpoint=(root / "vfms/moge/model.pt").resolve(), + cradio_dir=(root / "vfms/cradio/C-RADIOv4-SO400M").resolve(), + ) + + def validate(self) -> None: + missing = [ + p for p in ( + self.weights, self.foundation_stats, self.rvm_ckpt, + self.moge_checkpoint, self.cradio_dir, + ) + if not p.exists() + ] + if missing: + raise FileNotFoundError( + "missing runtime asset(s): " + ", ".join(str(p) for p in missing) + + "\nRun `python scripts/download_weights.py` to fetch them.") + + +@dataclass +class InferenceConfig: + hann_chunk: int = 80 + hann_stride: int = 40 + carry_hint: bool = True + hint_quality: float = 0.95 + carry_hint_quality: float = 0.95 + shape_quantum: int = 128 + model_scale: float = 1.0 + num_frames: int = -1 + start_mode: str = "begin" + seed: int = 0 + low_vram: bool = False + # Pick hann_chunk/hann_stride/low_vram per clip from free GPU memory. + auto_vram: bool = False + low_vram_foundation_chunk: int = 4 + low_vram_mlp_chunk_tokens: int = 65536 + low_vram_enc_frame_chunk: int = 1 + low_vram_swin_window_batch: int = 16 + low_vram_full_attn_query_chunk_tokens: int = 2048 + stage2_batch: int = 1 + outputs: set[str] = field(default_factory=lambda: {"alpha", "fg"}) + temp_dir: Path = Path(tempfile.gettempdir()) / "corridorkey_v2" + linear_brightness: float = 1.0 + linear_contrast: float = 1.0 + linear_contrast_pivot: float = 0.18 + frame_dir_fps: float = 24.0 + frame_dir_linear: bool = False + cutout_linear: bool = False + despill_strength: float = 0.5 + output_format: str = "video" + exr_dtype: str = "half" + + +# Peak GPU memory in GiB by megapixels of model input, measured at each Hann +# chunk size. Keys are low_vram False/True. +_VRAM_PEAK_GB = { + False: ((8, 24, 48, 80), ( + (0.98, (6.19, 7.52, 9.77, 12.89)), + (2.21, (7.50, 11.33, 18.66, 24.89)), + (3.93, (9.06, 14.78, 23.92, 33.19)), + (8.36, (13.94, 25.96, 45.15, 70.45)), + )), + True: ((1, 2, 4, 8, 24, 48, 80), ( + (0.98, (2.71, 2.78, 2.94, 3.25, 4.54, 6.57, 9.50)), + (2.21, (2.80, 2.97, 3.35, 4.12, 7.69, 13.20, 20.56)), + (3.93, (2.92, 3.23, 3.85, 5.09, 10.09, 17.57, 27.75)), + (8.36, (3.25, 3.92, 5.28, 8.38, 20.78, 39.66, 64.65)), + )), +} +_VRAM_MARGIN_SCALE = 1.1 +_VRAM_MARGIN_GB = 0.3 + + +def _interp(x: float, points) -> float: + """Piecewise-linear through (x, y) points, extrapolating past the ends.""" + points = list(points) + for (x0, y0), (x1, y1) in zip(points, points[1:]): + if x <= x1 or (x1, y1) == points[-1]: + return y0 + (y1 - y0) * (x - x0) / (x1 - x0) + return points[0][1] + + +def estimate_peak_vram_gb(megapixels: float, frames: int, low_vram: bool) -> float: + """Estimated peak GPU memory for a Hann window of `frames` frames.""" + chunks, rows = _VRAM_PEAK_GB[low_vram] + frames = max(frames, chunks[0]) + per_chunk = [ + max(_interp(megapixels, [(mp, peaks[i]) for mp, peaks in rows]), rows[0][1][i]) + for i in range(len(chunks)) + ] + return _interp(frames, zip(chunks, per_chunk)) + + +def select_vram_profile(shape_hw, num_frames: int, budget_gb: float): + """Largest Hann window that fits `budget_gb`, as (chunk, stride, low_vram, estimate). + + Bigger windows give better temporal consistency, and low-VRAM mode is + exact, so a larger low-VRAM window is preferred over a smaller normal one. + Falls back to a single-frame window when nothing is estimated to fit. + """ + megapixels = shape_hw[0] * shape_hw[1] / 1e6 + for chunk in reversed(_VRAM_PEAK_GB[True][0]): + for low_vram in (False, True): + if chunk not in _VRAM_PEAK_GB[low_vram][0]: + continue + estimate = estimate_peak_vram_gb(megapixels, min(chunk, num_frames), low_vram) + if estimate * _VRAM_MARGIN_SCALE + _VRAM_MARGIN_GB <= budget_gb: + return chunk, max(1, chunk // 2), low_vram, estimate + return 1, 1, True, estimate + + +def apply_auto_vram(cfg: InferenceConfig, shape_hw, num_frames: int) -> None: + free_bytes, _ = torch.cuda.mem_get_info() + # Memory this process already holds (the loaded model) is part of the budget. + budget_gb = (free_bytes + torch.cuda.memory_reserved()) / 2**30 + chunk, stride, low_vram, estimate = select_vram_profile(shape_hw, num_frames, budget_gb) + cfg.hann_chunk, cfg.hann_stride, cfg.low_vram = chunk, stride, low_vram + print(f" Auto VRAM: {budget_gb:.1f} GB available at {shape_hw[1]}x{shape_hw[0]} -> " + f"hann_chunk={chunk}, hann_stride={stride}, low_vram={low_vram} " + f"(estimated peak {estimate:.1f} GB)") + if estimate * _VRAM_MARGIN_SCALE + _VRAM_MARGIN_GB > budget_gb: + print(" [WARNING] Even a single-frame window may not fit this GPU at this " + "resolution. Inference may run out of memory.") + + +def prepare_runtime_environment(paths: RuntimePaths) -> None: + os.environ.setdefault("HF_HUB_OFFLINE", "1") + os.environ.setdefault("TRANSFORMERS_OFFLINE", "1") + moge_root = paths.package_root / "third_party/MoGe" + if str(moge_root) not in sys.path: + sys.path.insert(0, str(moge_root)) + + +def _resolve_package_path(paths: RuntimePaths, value: str | None) -> str | None: + if value is None: + return None + p = Path(value) + if not p.is_absolute(): + p = paths.package_root / p + return str(p.resolve()) + + +def load_runtime_checkpoint(path: Path) -> dict: + """Read a runtime checkpoint from .safetensors (preferred) or legacy .pt.""" + path = Path(path) + if path.suffix == ".safetensors": + from safetensors import safe_open + from safetensors.torch import load_file + with safe_open(str(path), framework="pt") as f: + header = f.metadata() or {} + return { + "format": header.get("format"), + "model_config": json.loads(header.get("model_config", "{}")), + "metadata": json.loads(header.get("metadata", "{}")), + "state_dict": load_file(str(path), device="cpu"), + } + return torch.load(path, map_location="cpu", weights_only=False, mmap=True) + + +def load_model(paths: RuntimePaths, device: str = "cuda") -> CorridorKeyV3: + paths.validate() + prepare_runtime_environment(paths) + ckpt = load_runtime_checkpoint(paths.weights) + if ckpt.get("format") != "corridorkey_v2_runtime_checkpoint": + raise RuntimeError(f"{paths.weights} is not a CorridorKey v2 runtime checkpoint") + cfg = dict(ckpt["model_config"]) + cfg["cradio_repo"] = str(paths.cradio_dir) + cfg["moge_checkpoint"] = str(paths.moge_checkpoint) + cfg["rvm_ckpt"] = str(paths.rvm_ckpt) + cfg["foundation_stats_path"] = str(paths.foundation_stats) + cfg["gradient_checkpoint"] = False + cfg["device"] = device + cfg["stage1_recurrent_64_sites"] = () + + print(f"Loading CorridorKey v2 runtime weights: {paths.weights}") + model = CorridorKeyV3(**cfg) + missing, unexpected = model.load_state_dict(ckpt["state_dict"], strict=False) + train_missing = [k for k in missing if not k.startswith("foundations.")] + train_unexpected = [k for k in unexpected if not k.startswith("foundations.")] + if train_missing or train_unexpected: + raise RuntimeError( + "runtime checkpoint did not match model: " + f"missing={train_missing[:8]} unexpected={train_unexpected[:8]}") + print(f" loaded weights; omitted frozen foundation tensors={len(missing)}") + model.eval() + del ckpt + return model + + +def configure_low_vram_inference(model, cfg: InferenceConfig): + model.stage1.low_vram_enc_frame_chunk = max(0, int(cfg.low_vram_enc_frame_chunk)) + for module in model.modules(): + name = module.__class__.__name__ + if name in ("SwiGLU", "FusionLinSwiglu"): + module.inference_chunk_tokens = max(0, int(cfg.low_vram_mlp_chunk_tokens)) + if name == "SwinAttention3D": + module.inference_window_batch = max(0, int(cfg.low_vram_swin_window_batch)) + if name == "CSwinAttention3D": + module.inference_strip_batch = max(0, int(cfg.low_vram_swin_window_batch)) + if name in ("FullSpatialAttention2D", "FullAttention3D"): + module.inference_query_chunk_tokens = max( + 0, int(cfg.low_vram_full_attn_query_chunk_tokens)) + + +def _build_foundation_inputs(rgb_bt: torch.Tensor): + B, T, _, H, W = rgb_bt.shape + flat = rgb_bt.reshape(B * T, 3, H, W) + rgb_radio = F.interpolate(flat, size=(H // 4, W // 4), mode="area") + H_moge, W_moge = (H * 7) // 32, (W * 7) // 32 + rgb_moge = F.interpolate(flat, size=(H_moge, W_moge), mode="area") + return ( + rgb_radio.reshape(B, T, 3, H // 4, W // 4), + rgb_moge.reshape(B, T, 3, H_moge, W_moge), + ) + + +def _next_or_none(iterator): + try: + return next(iterator) + except StopIteration: + return None + + +@torch.no_grad() +def precompute_rvm_to_memmap(model, source, meta, rvm_path: Path, + *, chunk: int, reader_kind: str): + assert model.foundations.rvm is not None + device = next(model.foundations.rvm.parameters()).device + T_all = int(meta["num_frames"]) + H, W = meta["shape_hw"] + H_e, W_e = H // 4, W // 4 + D = model.foundations.rvm.embed_dim + h_e, w_e = H_e // 16, W_e // 16 + print(f" RVM continuous prepass: T={T_all}, chunk={chunk}, " + f"input={W_e}x{H_e}, grid={w_e}x{h_e}") + out = np.memmap(rvm_path, dtype=np.float16, mode="w+", + shape=(T_all, D, h_e, w_e)) + mean = model.foundations.rvm_mean.view(1, 1, -1, 1, 1).float().cpu() + std = model.foundations.rvm_std.view(1, 1, -1, 1, 1).float().cpu().clamp(min=1e-6) + if reader_kind == "video": + chunks = iter_video_frame_chunks(str(source), meta, chunk_size=chunk, need_model=False) + else: + chunks = iter_frame_dir_chunks(str(source), meta, chunk_size=chunk, need_model=False) + state = None + with ThreadPoolExecutor(max_workers=1) as pool: + future = pool.submit(_next_or_none, chunks) + while True: + item = future.result() + if item is None: + break + future = pool.submit(_next_or_none, chunks) + t0, _model, display = item + Tc = display.shape[0] + rgb = display.unsqueeze(0).to(device, non_blocking=True).float() + rgb_rvm = F.interpolate( + rgb.reshape(Tc, 3, H, W), size=(H_e, W_e), + mode="bilinear", align_corners=False, + ).reshape(1, Tc, 3, H_e, W_e).clamp(0.0, 1.0) + with torch.autocast("cuda", dtype=torch.bfloat16): + feats, state = model.foundations.rvm(rgb_rvm, state=state) + feats_cpu = (feats.float().cpu() - mean) / std + out[t0:t0 + Tc] = feats_cpu[0].numpy().astype(np.float16) + del rgb, rgb_rvm, feats, feats_cpu, display + out.flush() + del out + torch.cuda.empty_cache() + + +@torch.no_grad() +def precompute_foundations_to_memmap(model, source, meta, prefix: Path, + *, chunk: int, reader_kind: str): + device = next(model.foundations.parameters()).device + T_all = int(meta["num_frames"]) + H, W = meta["shape_hw"] + h64, w64 = H // 64, W // 64 + C_cradio = int(getattr(model.stage1, "cradio_dim", 0) or 0) + C_moge = int(getattr(model.stage1, "moge_dim", 0) or 0) + if C_moge <= 0: + raise RuntimeError("MoGe is required for this model") + paths = { + "moge": str(prefix) + "_moge.f32", + "moge_shape": (T_all, C_moge, h64, w64), + } + moge_out = np.memmap(paths["moge"], dtype=np.float32, mode="w+", + shape=paths["moge_shape"]) + cradio_out = None + if C_cradio > 0: + paths["cradio"] = str(prefix) + "_cradio.f32" + paths["cradio_shape"] = (T_all, C_cradio, h64, w64) + cradio_out = np.memmap(paths["cradio"], dtype=np.float32, mode="w+", + shape=paths["cradio_shape"]) + print(f" frozen VFM prepass: T={T_all}, grid={w64}x{h64}, chunk={chunk}") + if reader_kind == "video": + chunks = iter_video_frame_chunks(str(source), meta, chunk_size=chunk, need_model=False) + else: + chunks = iter_frame_dir_chunks(str(source), meta, chunk_size=chunk, need_model=False) + with ThreadPoolExecutor(max_workers=1) as pool: + future = pool.submit(_next_or_none, chunks) + while True: + item = future.result() + if item is None: + break + future = pool.submit(_next_or_none, chunks) + t0, _model, display = item + Tc = display.shape[0] + rgb = display.unsqueeze(0).to(device, non_blocking=True) + rgb_radio, rgb_moge = _build_foundation_inputs(rgb) + with torch.autocast("cuda", dtype=torch.bfloat16): + ff = model.foundations( + rgb_cradio=rgb_radio.reshape(Tc, 3, *rgb_radio.shape[-2:]), + rgb_moge=rgb_moge.reshape(Tc, 3, *rgb_moge.shape[-2:]), + ) + moge_out[t0:t0 + Tc] = ff["moge"].float().cpu().numpy() + if cradio_out is not None: + cradio_out[t0:t0 + Tc] = ff["cradio"].float().cpu().numpy() + del rgb, rgb_radio, rgb_moge, ff, display + moge_out.flush() + del moge_out + if cradio_out is not None: + cradio_out.flush() + del cradio_out + torch.cuda.empty_cache() + return paths + + +def _phase1_setup(model, *, stage1_on_gpu: bool = True): + model.foundations.cuda() + # Low-VRAM mode precomputes foundation features first, so stage 1 waits on CPU. + if stage1_on_gpu: + model.stage1.cuda() + else: + model.stage1.cpu() + model.stage2.cpu() + torch.cuda.empty_cache() + + +def _phase2_setup(model): + model.foundations.cpu() + model.stage1.cpu() + torch.cuda.empty_cache() + model.stage2.cuda() + torch.cuda.synchronize() + + +@torch.no_grad() +def hann_stage1_stream(model, source, meta, *, reader_cls, reader_kind: str, + chunk: int, stride: int, baton_path: Path, + initial_hint, initial_hint_quality: float, + full_hint_video_path: str | None, + full_hint_quality: float, + rvm_path: Path | None, + foundation_paths, + carry_hint_from_aux: bool, + carry_hint_quality: float, + on_baton_frames): + device = next(model.stage1.parameters()).device + T_all = int(meta["num_frames"]) + H, W = meta["shape_hw"] + d_16 = model.stage1.d_16 + h16, w16 = H // 16, W // 16 + starts = window_starts(T_all, chunk, stride) + overlap = max(0, chunk - stride) + baton_fp16 = None + if on_baton_frames is None: + baton_fp16 = np.memmap(baton_path, dtype=np.float16, mode="w+", + shape=(T_all, d_16, h16, w16)) + live_acc = {} + live_w = {} + print(f" Hann stage1: chunk={chunk}, stride={stride}, " + f"windows={len(starts)}, baton_grid={w16}x{h16}") + + rvm_map = None + if rvm_path is not None: + D = model.foundations.rvm.embed_dim + rvm_map = np.memmap(rvm_path, dtype=np.float16, mode="r", + shape=(T_all, D, H // 64, W // 64)) + cradio_map = moge_map = None + if foundation_paths is not None: + moge_map = np.memmap(foundation_paths["moge"], dtype=np.float32, + mode="r", shape=tuple(foundation_paths["moge_shape"])) + if model.stage1.cradio_dim > 0: + cradio_map = np.memmap(foundation_paths["cradio"], dtype=np.float32, + mode="r", shape=tuple(foundation_paths["cradio_shape"])) + + reader = reader_cls(str(source), meta) + hint_reader = make_hint_reader(full_hint_video_path, meta) + carry_cache = {} + stage2_frame_cache = {} + + def flush_before(t_limit: int): + ready = [] + for k in sorted(k for k in live_acc if k < t_limit): + frame = (live_acc.pop(k) / max(live_w.pop(k), 1e-6)).astype(np.float16) + if on_baton_frames is None: + baton_fp16[k] = frame + else: + model_frame = stage2_frame_cache.pop(k) + ready.append((k, frame, model_frame)) + if ready: + on_baton_frames(ready) + + try: + for i, t0 in enumerate(starts): + t1 = min(t0 + chunk, T_all) + Tc = t1 - t0 + model_chunk, display_chunk, review_chunk = reader.window(t0, t1) + hint_cpu = torch.zeros(1, Tc, 1, H, W, dtype=torch.bfloat16) + hint_quality_cpu = torch.zeros(1, Tc, dtype=torch.bfloat16) + hint_present = torch.zeros(1, Tc, dtype=torch.bool) + if hint_reader is not None: + hint_cpu[0] = hint_reader.window(t0, t1) + hint_quality_cpu.fill_(float(full_hint_quality)) + hint_present.fill_(True) + for j, t_abs in enumerate(range(t0, t1)): + if t_abs in carry_cache: + hint_cpu[0, j] = carry_cache[t_abs] + hint_quality_cpu[0, j] = float(carry_hint_quality) + hint_present[0, j] = True + if initial_hint is not None and t0 == 0: + hint_cpu[0, 0] = initial_hint.to(dtype=hint_cpu.dtype) + hint_quality_cpu[0, 0] = float(initial_hint_quality) + hint_present[0, 0] = True + if not bool(hint_present.any().item()): + hint_quality_cpu.fill_(-1.0) + + if on_baton_frames is not None: + for j, t_abs in enumerate(range(t0, t1)): + stage2_frame_cache[t_abs] = model_chunk[j] + + rgb_c = model_chunk.unsqueeze(0).to(device, non_blocking=True) + hint_c = hint_cpu.to(device, non_blocking=True) + hint_quality_c = hint_quality_cpu.to(device, non_blocking=True) + ff = rgb_radio = rgb_moge = None + if foundation_paths is None: + rgb_found = display_chunk.unsqueeze(0).to(device, non_blocking=True) + rgb_radio, rgb_moge = _build_foundation_inputs(rgb_found) + with torch.autocast("cuda", dtype=torch.bfloat16): + ff = model.foundations( + rgb_cradio=rgb_radio.reshape(Tc, 3, *rgb_radio.shape[-2:]), + rgb_moge=rgb_moge.reshape(Tc, 3, *rgb_moge.shape[-2:]), + ) + cradio_feat = ff["cradio"].reshape(1, Tc, -1, H // 64, W // 64) + moge_feat = ff["moge"].reshape(1, Tc, -1, H // 64, W // 64) + else: + moge_feat = torch.from_numpy(np.asarray(moge_map[t0:t1]).copy()) + moge_feat = moge_feat.unsqueeze(0).to(device, non_blocking=True).float() + cradio_feat = torch.from_numpy(np.asarray(cradio_map[t0:t1]).copy()) + cradio_feat = cradio_feat.unsqueeze(0).to(device, non_blocking=True).float() + rvm_feat = None + if rvm_map is not None: + rvm_feat = torch.from_numpy(np.asarray(rvm_map[t0:t1]).copy()) + rvm_feat = rvm_feat.unsqueeze(0).to(device, non_blocking=True).float() + with torch.autocast("cuda", dtype=torch.bfloat16): + s1 = model.stage1( + rgb_c, hint_c, moge_feat, cradio_feat=cradio_feat, + rvm_feat=rvm_feat, t_stride=1.0, + hint_quality=hint_quality_c, + ) + baton_gpu = s1["baton"][0].float() + if carry_hint_from_aux: + aux_alpha = s1["aux"][0, :, 0:1].float() + aux_full = F.interpolate( + aux_alpha, size=(H, W), mode="bilinear", + align_corners=False).clamp(0.0, 1.0) + aux_cpu = aux_full.to("cpu", dtype=torch.bfloat16) + for j, t_abs in enumerate(range(t0, t1)): + carry_cache[t_abs] = aux_cpu[j].clone() + next_start = starts[i + 1] if i + 1 < len(starts) else T_all + for old_t in [k for k in carry_cache if k < next_start]: + del carry_cache[old_t] + del aux_alpha, aux_full, aux_cpu + w = taper_weights(Tc, overlap, is_first=(i == 0), + is_last=(i == len(starts) - 1), + device="cpu", dtype=torch.float32) + weighted = (baton_gpu * w.to(device).view(Tc, 1, 1, 1)).to("cpu").numpy() + w_np = w.numpy() + for j, t_abs in enumerate(range(t0, t1)): + if t_abs not in live_acc: + live_acc[t_abs] = weighted[j].copy() + live_w[t_abs] = float(w_np[j]) + else: + live_acc[t_abs] += weighted[j] + live_w[t_abs] += float(w_np[j]) + next_start = starts[i + 1] if i + 1 < len(starts) else T_all + del s1, baton_gpu, weighted, ff, cradio_feat, moge_feat, rvm_feat + del rgb_c, hint_c, hint_quality_c, hint_cpu, hint_quality_cpu + del model_chunk, display_chunk, review_chunk + flush_before(next_start) + print(f" window {i + 1}/{len(starts)} [{t0}:{t1}]", flush=True) + torch.cuda.synchronize() + flush_before(T_all + 1) + finally: + reader.close() + if hint_reader is not None: + hint_reader.close() + + if baton_fp16 is not None: + baton_fp16.flush() + del baton_fp16, live_acc, live_w, rvm_map, cradio_map, moge_map + torch.cuda.empty_cache() + + +def _crop_native_torch(x: torch.Tensor, meta: dict) -> torch.Tensor: + h, w = meta["model_native_hw"] + pad_l, _, pad_t, _ = meta["pad"] + return x[..., pad_t:pad_t + h, pad_l:pad_l + w] + + +def _resize_output_torch(x: torch.Tensor, meta: dict) -> torch.Tensor: + if tuple(meta["model_native_hw"]) == tuple(meta["output_hw"]): + return x + raise RuntimeError( + "optimized runtime render path requires native model_scale=1.0; " + "non-native scaling would need explicit Lanczos output rendering") + + +def _linear_adjust_inverse_torch(linear: torch.Tensor, meta: dict) -> torch.Tensor: + gain = float(meta.get("linear_brightness", 1.0)) + contrast = float(meta.get("linear_contrast", 1.0)) + if abs(gain - 1.0) <= 1e-6 and abs(contrast - 1.0) <= 1e-6: + return linear + pivot = float(meta.get("linear_contrast_pivot", 0.18)) + return (linear / gain - pivot) / contrast + pivot + + +def _despill_average_srgb_torch(fg: torch.Tensor, strength: float = 0.5) -> torch.Tensor: + if strength <= 0.0: + return fg + r = fg[:, 0:1] + g = fg[:, 1:2] + b = fg[:, 2:3] + spill = (g - (r + b) * 0.5).clamp(min=0.0) + full = torch.cat([r + spill * 0.5, g - spill, b + spill * 0.5], dim=1) + if strength >= 1.0: + return full + return fg * (1.0 - strength) + full * strength + + +def _srgb_to_bgr_uint16_cpu(x: torch.Tensor) -> np.ndarray: + x = x.clamp(0.0, 1.0) + x = x[:, [2, 1, 0]].permute(0, 2, 3, 1).contiguous().cpu().numpy() + return (x * 65535.0).astype(np.uint16) + + +def _srgb_to_bgra_uint16_cpu(rgb: torch.Tensor, alpha: torch.Tensor) -> np.ndarray: + rgb = rgb.clamp(0.0, 1.0) + alpha = alpha.clamp(0.0, 1.0) + bgra = torch.cat([rgb[:, [2, 1, 0]], alpha], dim=1) + bgra = bgra.permute(0, 2, 3, 1).contiguous().cpu().numpy() + return (bgra * 65535.0).astype(np.uint16) + + +def _chw_to_bgr_f32_cpu(x: torch.Tensor) -> np.ndarray: + return (x[:, [2, 1, 0]].permute(0, 2, 3, 1) + .contiguous().cpu().numpy().astype(np.float32, copy=False)) + + +def _chw_to_bgra_f32_cpu(rgb: torch.Tensor, alpha: torch.Tensor) -> np.ndarray: + bgra = torch.cat([rgb[:, [2, 1, 0]], alpha], dim=1) + return (bgra.permute(0, 2, 3, 1) + .contiguous().cpu().numpy().astype(np.float32, copy=False)) + + +def _render_predictions_exr_torch(meta: dict, outputs: set[str], + checker: torch.Tensor, + logits: torch.Tensor) -> dict[str, np.ndarray]: + """Render a stage-2 batch as unclamped scene-linear float32 NHWC arrays. + + HDR contract: `fg`, `cutout`, and `checker` color stays in decoded linear + light with no [0, 1] clamp and no quantization, so values above 1.0 (and + negative values the model may emit) survive into the EXR files. Alpha is a + matte and stays clamped to [0, 1]. + """ + alpha_model = logits[:, 0:1].float() + pa_model = _crop_native_torch(alpha_model, meta).clamp(0.0, 1.0) + despill_strength = float(meta.get("despill_strength", 0.5)) + + pa = _resize_output_torch(pa_model, meta).clamp(0.0, 1.0) + rendered: dict[str, np.ndarray] = {} + if "alpha" in outputs: + rendered["alpha"] = (pa.permute(0, 2, 3, 1) + .contiguous().cpu().numpy() + .astype(np.float32, copy=False)) + if not (outputs & {"fg", "cutout", "checker"}): + return rendered + + fg_model = logits[:, 1:4].float() + pf_model = _crop_native_torch(fg_model, meta) + pf_linear = _linear_adjust_inverse_torch(cu.asinh_to_linear(pf_model), meta) + pf_linear = _resize_output_torch(pf_linear, meta) + if "fg" in outputs: + rendered["fg"] = _chw_to_bgr_f32_cpu(pf_linear) + if "cutout" in outputs: + pf_linear_despilled = _despill_average_srgb_torch( + pf_linear, strength=despill_strength) + rendered["cutout"] = _chw_to_bgra_f32_cpu(pf_linear_despilled, pa) + if "checker" in outputs: + checker_b = checker.to(device=pf_linear.device, + dtype=pf_linear.dtype).unsqueeze(0) + bg_lin = cu.srgb_to_linear(checker_b) + pf_despill = _despill_average_srgb_torch( + pf_linear, strength=despill_strength) + comp_lin = pf_despill * pa + bg_lin * (1.0 - pa) + rendered["checker"] = _chw_to_bgr_f32_cpu(comp_lin) + return rendered + + +def _render_predictions_torch(meta: dict, outputs: set[str], + checker: torch.Tensor, + logits: torch.Tensor) -> dict[str, np.ndarray]: + """Render a stage-2 batch on GPU, returning uint16 BGR/BGRA NHWC arrays. + + This removes the previous per-frame numpy HDR decode, sRGB encode, + composition, channel shuffle, and uint8 conversion hot path. + """ + alpha_model = logits[:, 0:1].float() + pa_model = _crop_native_torch(alpha_model, meta).clamp(0.0, 1.0) + despill_strength = float(meta.get("despill_strength", 0.5)) + + pa = _resize_output_torch(pa_model, meta).clamp(0.0, 1.0) + rendered: dict[str, np.ndarray] = {} + if "alpha" in outputs: + rendered["alpha"] = _srgb_to_bgr_uint16_cpu(pa.expand(-1, 3, -1, -1)) + if not (outputs & {"fg", "cutout", "checker"}): + return rendered + + fg_model = logits[:, 1:4].float() + pf_model = _crop_native_torch(fg_model, meta) + pf_linear = _linear_adjust_inverse_torch(cu.asinh_to_linear(pf_model), meta) + pf_linear_pos = pf_linear.clamp(min=0.0) + pf = _resize_output_torch(cu.linear_to_srgb(pf_linear_pos), meta).clamp(0.0, 1.0) + if "fg" in outputs: + rendered["fg"] = _srgb_to_bgr_uint16_cpu(pf) + if "cutout" in outputs: + pf_linear_despilled = _despill_average_srgb_torch(pf_linear_pos, strength=despill_strength) + if meta.get("cutout_linear", False): + pf_linear_despilled_resized = _resize_output_torch(pf_linear_despilled, meta).clamp(0.0, 1.0) + rendered["cutout"] = _srgb_to_bgra_uint16_cpu(pf_linear_despilled_resized, pa) + else: + pf_despill = _despill_average_srgb_torch(pf, strength=despill_strength) + rendered["cutout"] = _srgb_to_bgra_uint16_cpu(pf_despill, pa) + if "checker" in outputs: + checker_b = checker.to(device=pf.device, dtype=pf.dtype).unsqueeze(0) + bg_lin = cu.srgb_to_linear(checker_b) + alpha = pa + pf_despill = _despill_average_srgb_torch(pf, strength=despill_strength) + comp_despill_lin = ( + cu.srgb_to_linear(pf_despill) * alpha + bg_lin * (1.0 - alpha)) + comp_despill = cu.linear_to_srgb( + comp_despill_lin.clamp(min=0.0)).clamp(0.0, 1.0) + rendered["checker"] = _srgb_to_bgr_uint16_cpu(comp_despill) + return rendered + + +def run_inference(model, input_path: str, output_dir: str, cfg: InferenceConfig, + writer_cfg: WriterConfig, *, initial_hint_path: str | None, + hint_video_path: str | None): + input_path = str(input_path) + output_dir = str(output_dir) + cfg.temp_dir.mkdir(parents=True, exist_ok=True) + os.makedirs(output_dir, exist_ok=True) + is_dir = os.path.isdir(input_path) + reader_kind = "frame_dir" if is_dir else "video" + reader_cls = SequentialFrameDirWindowReader if is_dir else SequentialVideoWindowReader + if is_dir: + meta = probe_frame_dir_meta( + input_path, num_frames=cfg.num_frames, + shape_quantum=cfg.shape_quantum, fps=cfg.frame_dir_fps, + exr_linear=cfg.frame_dir_linear, model_scale=cfg.model_scale) + if (not cfg.frame_dir_linear + and meta["frame_paths"] + and meta["frame_paths"][0].lower().endswith(".exr")): + print(" [WARNING] EXR input frames are being treated as " + "display-coded and clipped to [0, 1]. For scene-linear HDR " + "EXR plates, pass --frame_dir_linear.") + else: + meta = probe_video_meta( + input_path, num_frames=cfg.num_frames, + shape_quantum=cfg.shape_quantum, native_resolution=True, + start_mode=cfg.start_mode, seed=cfg.seed, + model_scale=cfg.model_scale, + linear_brightness=cfg.linear_brightness, + linear_contrast=cfg.linear_contrast, + linear_contrast_pivot=cfg.linear_contrast_pivot) + T_all = int(meta["num_frames"]) + H, W = meta["shape_hw"] + if cfg.auto_vram: + apply_auto_vram(cfg, (H, W), T_all) + H_out, W_out = meta["output_hw"] + meta["cutout_linear"] = cfg.cutout_linear + meta["despill_strength"] = cfg.despill_strength + fps = float(meta["native_fps"]) + clip_name = raw_clip_name(input_path) + checker = make_checker(H_out, W_out) + checker_t = torch.from_numpy(checker) + writers = {} + suffix = { + "alpha": "alpha", + "fg": "fg", + "checker": f"checker_comp_despill{int(cfg.despill_strength * 100):03d}", + "cutout": "cutout", + } + try: + use_exr = cfg.output_format == "exr" + if cfg.output_format not in ("video", "exr"): + raise ValueError(f"unsupported output format: {cfg.output_format}") + for name in sorted(cfg.outputs): + if name not in suffix: + raise ValueError(f"unsupported output choice: {name}") + if use_exr: + channels = 4 if name == "cutout" else (1 if name == "alpha" else 3) + writers[name] = ExrSequenceWriter( + os.path.join(output_dir, f"{clip_name}_{suffix[name]}_exr"), + fps, (W_out, H_out), channels=channels, + dtype=cfg.exr_dtype, frame_name=suffix[name]) + else: + ext = ".mov" if name == "cutout" else ".mp4" + channels = 4 if name == "cutout" else 3 + color_trc = "linear" if (name == "cutout" and cfg.cutout_linear) else "iec61966-2-1" + writers[name] = ThreadedVideoWriter( + os.path.join(output_dir, f"{clip_name}_{suffix[name]}{ext}"), + fps, (W_out, H_out), writer_cfg, channels=channels, color_trc=color_trc) + + # Write settings log file in the output folder + settings_path = os.path.join(output_dir, f"{clip_name}_settings.txt") + try: + with open(settings_path, "w") as sf: + sf.write(f"CorridorKey v2 Inference Settings Log\n") + sf.write(f"====================================\n") + sf.write(f"Input: {input_path}\n") + sf.write(f"Output Directory: {output_dir}\n") + sf.write(f"Resolution: {W_out}x{H_out}\n") + sf.write(f"FPS: {fps}\n") + sf.write(f"Outputs: {', '.join(sorted(cfg.outputs))}\n") + sf.write(f"Output Format: {cfg.output_format}\n") + if cfg.output_format == "exr": + sf.write(f"EXR Pixel Type: {cfg.exr_dtype}\n") + sf.write(f"Hint First Frame: {initial_hint_path}\n") + sf.write(f"Hint Video: {hint_video_path}\n") + sf.write(f"Hint Quality (Trust): {cfg.hint_quality}\n") + sf.write(f"Carry Hint: {cfg.carry_hint}\n") + sf.write(f"Carry Hint Quality: {cfg.carry_hint_quality}\n") + sf.write(f"Despill Strength: {cfg.despill_strength}\n") + sf.write(f"Cutout Linear: {cfg.cutout_linear}\n") + sf.write(f"Frame Dir Linear: {cfg.frame_dir_linear}\n") + sf.write(f"Hann Chunk: {cfg.hann_chunk}\n") + sf.write(f"Hann Stride: {cfg.hann_stride}\n") + sf.write(f"Low VRAM: {cfg.low_vram}\n") + sf.write(f"Model Scale: {cfg.model_scale}\n") + sf.write(f"Num Frames: {cfg.num_frames}\n") + sf.write(f"Start Mode: {cfg.start_mode}\n") + sf.write(f"Seed: {cfg.seed}\n") + sf.write(f"Linear Brightness: {cfg.linear_brightness}\n") + sf.write(f"Linear Contrast: {cfg.linear_contrast}\n") + sf.write(f"Linear Contrast Pivot: {cfg.linear_contrast_pivot}\n") + except Exception as e: + print(f"[WARNING] Failed to write settings file: {e}") + + initial_hint = ( + load_initial_hint_frame(initial_hint_path, meta) + if none_path(initial_hint_path) is not None else None + ) + _phase1_setup(model, stage1_on_gpu=not cfg.low_vram) + if cfg.low_vram: + configure_low_vram_inference(model, cfg) + rvm_path = cfg.temp_dir / f"{clip_name}_rvm_feat.f16" + baton_path = cfg.temp_dir / f"{clip_name}_baton.f16" + foundation_prefix = cfg.temp_dir / f"{clip_name}_foundation" + for p in ( + rvm_path, baton_path, Path(str(foundation_prefix) + "_cradio.f32"), + Path(str(foundation_prefix) + "_moge.f32"), + ): + safe_remove(p) + if model.foundations.rvm is not None: + precompute_rvm_to_memmap( + model, input_path, meta, rvm_path, + chunk=max(1, min(int(cfg.hann_chunk), 8)), + reader_kind=reader_kind) + rvm_feat_path = rvm_path + else: + rvm_feat_path = None + foundation_paths = None + if cfg.low_vram: + foundation_paths = precompute_foundations_to_memmap( + model, input_path, meta, foundation_prefix, + chunk=max(1, int(cfg.low_vram_foundation_chunk)), + reader_kind=reader_kind) + model.foundations.cpu() + model.stage2.cpu() + model.stage1.cuda() + torch.cuda.empty_cache() + else: + model.stage2.cuda() + model.stage2.eval() + torch.cuda.empty_cache() + + stage2_batch = max(1, int(cfg.stage2_batch)) + + def write_prediction_batches(items): + device = next(model.stage2.parameters()).device + for offset in range(0, len(items), stage2_batch): + sub = items[offset:offset + stage2_batch] + rgb_t = torch.stack([item[2] for item in sub], dim=0) + rgb_t = rgb_t.unsqueeze(0).to(device, non_blocking=True) + baton_np = np.stack([item[1] for item in sub], axis=0) + baton_t = torch.from_numpy(baton_np).unsqueeze(0) + baton_t = baton_t.to(device, non_blocking=True).float() + with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): + logits = model.stage2(rgb_t, baton_t) + if isinstance(logits, tuple): + logits = logits[0] + render_fn = (_render_predictions_exr_torch if use_exr + else _render_predictions_torch) + rendered = render_fn(meta, cfg.outputs, checker_t, logits[0]) + for name, frames in rendered.items(): + writers[name].write_many(frames) + del logits, baton_t, rgb_t, baton_np, rendered + + hann_stage1_stream( + model, input_path, meta, reader_cls=reader_cls, + reader_kind=reader_kind, chunk=cfg.hann_chunk, stride=cfg.hann_stride, + baton_path=baton_path, initial_hint=initial_hint, + initial_hint_quality=cfg.hint_quality, + full_hint_video_path=hint_video_path, + full_hint_quality=cfg.hint_quality, + rvm_path=rvm_feat_path, foundation_paths=foundation_paths, + carry_hint_from_aux=cfg.carry_hint, + carry_hint_quality=cfg.carry_hint_quality, + on_baton_frames=None if cfg.low_vram else write_prediction_batches) + + if cfg.low_vram: + _phase2_setup(model) + baton = np.memmap( + baton_path, dtype=np.float16, mode="r", + shape=(T_all, model.stage1.d_16, H // 16, W // 16)) + reader = reader_cls(input_path, meta) + try: + for t0 in range(0, T_all, stage2_batch): + t1 = min(T_all, t0 + stage2_batch) + model_chunk, _display_chunk, _review_chunk = reader.window(t0, t1) + items = [ + (t, np.asarray(baton[t]).copy(), model_chunk[t - t0]) + for t in range(t0, t1) + ] + write_prediction_batches(items) + del model_chunk, _display_chunk, _review_chunk, items + finally: + reader.close() + del baton + finally: + for writer in writers.values(): + writer.release() + safe_remove(cfg.temp_dir / f"{raw_clip_name(input_path)}_rvm_feat.f16") + safe_remove(cfg.temp_dir / f"{raw_clip_name(input_path)}_baton.f16") + safe_remove(cfg.temp_dir / f"{raw_clip_name(input_path)}_foundation_cradio.f32") + safe_remove(cfg.temp_dir / f"{raw_clip_name(input_path)}_foundation_moge.f32") + torch.cuda.empty_cache() + print(f" wrote {len(writers)} output video(s) for {clip_name} " + f"({T_all} frames @ {W_out}x{H_out})") diff --git a/corridorkey/rvm.py b/corridorkey/rvm.py new file mode 100644 index 0000000000000000000000000000000000000000..09eb9720317152987469bad3bce4258bdc38c79c --- /dev/null +++ b/corridorkey/rvm.py @@ -0,0 +1,562 @@ +"""RVM (Recurrent Video MAE) — PyTorch port of the representations4d RVM. + +Architecture (from +https://github.com/google-deepmind/representations4d/blob/main/colabs/rvm_inference_demo.ipynb): + VideoSiamMAE(frame, state): + 1. Patchify (2D 16×16 conv, T-kernel=1) + 2. + 2D sinusoidal posenc, bicubic-resized from base 16×16 + 3. Prepend learned CLS token. + 4. Run a plain ViT encoder (12 PreNorm blocks). + 5. Final LayerNorm. + 6. Feed to GatedTransformerCore: + - GRU gate: update = σ(W_iu·x + W_su·s); reset = σ(W_ir·x + W_sr·s) + - h = CrossAttnTransformer(queries=x, kv=reset·LN_noBias(s)) + - state_new = (1 - update)·s + update·h + - return state_new, state_new + 7. Features per frame = state at that time step (including CLS slot). + +This port drops the MAE pretrain decoder — we only need the representation +path (encoder + rnn_core), which is 33M of the 67M param checkpoint for S. + +Input contract: + video: (B, T, 3, H, W) fp32 in [0, 1]. + IMPORTANT: RVM was pretrained on raw [0, 1] video frames; no imagenet + mean/std normalization applied. + +Output: + features: (B, T, D, h, w) where h=H/16, w=W/16 and D=embed_dim. + final_state: (B, 1+h·w, D) — for continuing stateful inference across + clips (we carry this across Hann windows at inference time). +""" +from typing import Optional, Tuple + +import math + +import numpy as np +import torch +import torch.nn as nn +import torch.nn.functional as F + + +# --------------------------------------------------------------------------- +# Model configs (from ckpt inspection) +# --------------------------------------------------------------------------- + +RVM_CFG = { + # RVM-S — ViT-S encoder (D=384), 4 xa_blocks in rnn_core with 8×48 heads. + # Verified by inspection of pretrain_rvm_small16_256_204031069.npz. + 'small': dict( + embed_dim=384, + enc_num_layers=12, enc_num_heads=6, enc_head_dim=64, + enc_mlp_hidden=1536, + rnn_num_xa_blocks=4, rnn_num_heads=8, rnn_head_dim=48, + rnn_mlp_hidden=2048, + ), + # RVM-B — ViT-B encoder (D=768), 4 xa_blocks in rnn_core with 12×64 heads. + # Verified by inspection of pretrain_rvm_base16_256_203916225.npz. + 'base': dict( + embed_dim=768, + enc_num_layers=12, enc_num_heads=12, enc_head_dim=64, + enc_mlp_hidden=3072, + rnn_num_xa_blocks=4, rnn_num_heads=12, rnn_head_dim=64, + rnn_mlp_hidden=3072, + ), +} + +PATCH_SIZE = 16 +BASE_TOKEN_H = 16 +BASE_TOKEN_W = 16 + + +# --------------------------------------------------------------------------- +# 2D sinusoidal positional encoding (bicubic-resizable from base 16×16) +# --------------------------------------------------------------------------- + +def _sincos_table_1d(n: int, d: int, dtype=torch.float32) -> torch.Tensor: + """MAE-style 1D sinusoidal table of shape (n, d). + angles[i, j] = i / 10000^(2·⌊j/2⌋ / d). Even j → sin, odd j → cos.""" + t = torch.zeros(n, d, dtype=dtype) + pos = torch.arange(n, dtype=dtype)[:, None] + j = torch.arange(d, dtype=dtype)[None, :] + angle = pos / torch.pow(10000.0, 2.0 * (j // 2) / d) + t[:, 0::2] = torch.sin(angle[:, 0::2]) + t[:, 1::2] = torch.cos(angle[:, 1::2]) + return t + + +def _build_base_posenc(d: int, + base_h: int = BASE_TOKEN_H, + base_w: int = BASE_TOKEN_W, + dtype=torch.float32) -> torch.Tensor: + """(base_h, base_w, d) sinusoidal positional encoding, per the upstream + `SincosPosEmb` construction: + raw = sincos_1d(base_h·base_w, d) → reshape to (base_h, base_w, d) + """ + raw = _sincos_table_1d(base_h * base_w, d, dtype=dtype) + return raw.view(base_h, base_w, d) + + +def _resize_posenc(base: torch.Tensor, target_h: int, target_w: int) -> torch.Tensor: + """Bicubic-resize a (base_h, base_w, D) posenc table to (target_h, target_w, D).""" + if base.shape[0] == target_h and base.shape[1] == target_w: + return base + # (H, W, D) → (1, D, H, W) for F.interpolate + x = base.permute(2, 0, 1).unsqueeze(0) + x = F.interpolate(x, size=(target_h, target_w), mode='bicubic', align_corners=False) + # Back to (target_h, target_w, D) + return x.squeeze(0).permute(1, 2, 0).contiguous() + + +# --------------------------------------------------------------------------- +# Attention / MLP / blocks (matching JAX layout for clean weight conversion) +# --------------------------------------------------------------------------- + +class _MHA(nn.Module): + """Multi-head dot-product attention. + + Matches JAX `ImprovedMultiHeadDotProductAttention` layout: separate + `query`, `key`, `value`, and `out` linears, each with bias. + + Allows cross-attention via (q, k, v) with k=v optional. head_dim may + differ from embed_dim//num_heads (e.g. rnn_core uses 8×48=384 on D=384). + """ + + def __init__(self, d_model: int, num_heads: int, head_dim: int): + super().__init__() + self.d_model = d_model + self.num_heads = num_heads + self.head_dim = head_dim + self.inner_dim = num_heads * head_dim + self.query = nn.Linear(d_model, self.inner_dim, bias=True) + self.key = nn.Linear(d_model, self.inner_dim, bias=True) + self.value = nn.Linear(d_model, self.inner_dim, bias=True) + self.out = nn.Linear(self.inner_dim, d_model, bias=True) + + def forward(self, q_in: torch.Tensor, + k_in: Optional[torch.Tensor] = None, + v_in: Optional[torch.Tensor] = None) -> torch.Tensor: + if k_in is None: + k_in = q_in + if v_in is None: + v_in = k_in + q = self.query(q_in) # (..., Lq, H·D) + k = self.key(k_in) # (..., Lk, H·D) + v = self.value(v_in) # (..., Lk, H·D) + # Reshape per head: (..., L, H, D) → (..., H, L, D). + def split_heads(x): + *lead, L, _ = x.shape + return x.view(*lead, L, self.num_heads, self.head_dim).transpose(-3, -2) + q = split_heads(q); k = split_heads(k); v = split_heads(v) + # Use flash SDPA for speed / memory. + attn = F.scaled_dot_product_attention(q, k, v, dropout_p=0.0, is_causal=False) + # (..., H, Lq, D) → (..., Lq, H, D) → (..., Lq, H·D) + attn = attn.transpose(-3, -2).contiguous() + attn = attn.reshape(*attn.shape[:-2], self.inner_dim) + return self.out(attn) + + +class _TransformerMLP(nn.Module): + """GELU MLP: Dense_in → GELU → Dense_out. Matches JAX layout.""" + + def __init__(self, d_model: int, hidden: int): + super().__init__() + self.dense_in = nn.Linear(d_model, hidden, bias=True) + self.dense_out = nn.Linear(hidden, d_model, bias=True) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.dense_out(F.gelu(self.dense_in(x))) + + +class _PreNormBlock(nn.Module): + """Encoder block: pre-LN MHA + pre-LN MLP, both residual.""" + + def __init__(self, d_model: int, num_heads: int, head_dim: int, mlp_hidden: int): + super().__init__() + self.attention_norm = nn.LayerNorm(d_model) + self.attention = _MHA(d_model, num_heads, head_dim) + self.mlp_norm = nn.LayerNorm(d_model) + self.mlp = _TransformerMLP(d_model, mlp_hidden) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = x + self.attention(self.attention_norm(x)) + x = x + self.mlp(self.mlp_norm(x)) + return x + + +class _CrossAttentionBlock(nn.Module): + """Cross-attention block used inside GatedTransformerCore. + + Block order matches the upstream notebook's `CrossAttentionBlock.__call__`: + x = x + ca_attention(ca_norm(x), kv=kv) # cross-attn over kv + x = x + mlp(mlp_norm(x)) + x = x + attention(attention_norm(x)) # self-attn + """ + + def __init__(self, d_model: int, num_heads: int, head_dim: int, mlp_hidden: int): + super().__init__() + self.ca_attention_norm = nn.LayerNorm(d_model) + self.ca_attention = _MHA(d_model, num_heads, head_dim) + self.mlp_norm = nn.LayerNorm(d_model) + self.mlp = _TransformerMLP(d_model, mlp_hidden) + self.attention_norm = nn.LayerNorm(d_model) + self.attention = _MHA(d_model, num_heads, head_dim) + + def forward(self, x: torch.Tensor, kv: torch.Tensor) -> torch.Tensor: + x = x + self.ca_attention(self.ca_attention_norm(x), k_in=kv, v_in=kv) + x = x + self.mlp(self.mlp_norm(x)) + x = x + self.attention(self.attention_norm(x)) + return x + + +class _CrossAttentionTransformer(nn.Module): + """Stack of xa_blocks + final output_norm.""" + + def __init__(self, d_model: int, num_heads: int, head_dim: int, + mlp_hidden: int, num_blocks: int): + super().__init__() + self.xa_blocks = nn.ModuleList([ + _CrossAttentionBlock(d_model, num_heads, head_dim, mlp_hidden) + for _ in range(num_blocks) + ]) + self.output_norm = nn.LayerNorm(d_model) + + def forward(self, x: torch.Tensor, kv: torch.Tensor) -> torch.Tensor: + for blk in self.xa_blocks: + x = blk(x, kv) + return self.output_norm(x) + + +class _GatedTransformerCore(nn.Module): + """GRU-gated cross-attention between current frame tokens and recurrent state. + + update = σ(W_iu·x + W_su·s) + reset = σ(W_ir·x + W_sr·s) + h = CrossAttnTransformer(queries=x, kv = reset · state_LN(s)) + s' = (1 - update)·s + update·h + return s', s' + + All four gate linears are bias-free (matches upstream NPZ — no `bias` key + for input_update/input_reset/state_update/state_reset). + + `state_layer_norm` has *only* the scale parameter (no bias), per the + upstream `nn.LayerNorm(use_bias=False)`. PyTorch LayerNorm has bias by + default; we zero-and-freeze the bias there. + """ + + def __init__(self, d_model: int, num_heads: int, head_dim: int, + mlp_hidden: int, num_xa_blocks: int): + super().__init__() + self.d_model = d_model + self.input_update = nn.Linear(d_model, d_model, bias=False) + self.input_reset = nn.Linear(d_model, d_model, bias=False) + self.state_update = nn.Linear(d_model, d_model, bias=False) + self.state_reset = nn.Linear(d_model, d_model, bias=False) + # bias-free LN. PyTorch LayerNorm always has a bias Parameter; we + # just keep it at zero (and upstream ckpt has no corresponding bias). + self.state_layer_norm = nn.LayerNorm(d_model, eps=1e-4) + nn.init.zeros_(self.state_layer_norm.bias) + self.state_layer_norm.bias.requires_grad = False + + self.transformer = _CrossAttentionTransformer( + d_model=d_model, num_heads=num_heads, head_dim=head_dim, + mlp_hidden=mlp_hidden, num_blocks=num_xa_blocks, + ) + + def forward(self, inputs: torch.Tensor, state: torch.Tensor + ) -> Tuple[torch.Tensor, torch.Tensor]: + """inputs: (B, N, D) current encoded frame tokens. state: (B, N, D). + Returns (features, new_state).""" + update = torch.sigmoid(self.input_update(inputs) + self.state_update(state)) + reset = torch.sigmoid(self.input_reset(inputs) + self.state_reset(state)) + gated_kv = reset * self.state_layer_norm(state) + h = self.transformer(inputs, gated_kv) + out = (1.0 - update) * state + update * h + return out, out + + +# --------------------------------------------------------------------------- +# Main model +# --------------------------------------------------------------------------- + + +class RVMVideoSiamMAE(nn.Module): + """RVM encoder + rnn_core (MAE decoder dropped). + + Processes video frame-by-frame. Encoder is fully batchable over B·T; + only the rnn_core is sequential across T (carrying state). + """ + + def __init__(self, variant: str = 'small'): + super().__init__() + cfg = RVM_CFG[variant] + self.variant = variant + self.embed_dim = cfg['embed_dim'] + + # Tokenizer: 2D patch conv + learned CLS token + sincos posenc. + self.patch_embed = nn.Conv2d(3, self.embed_dim, + kernel_size=PATCH_SIZE, + stride=PATCH_SIZE, bias=True) + self.cls_token = nn.Parameter(torch.zeros(1, self.embed_dim)) + + # Pre-computed base posenc table (non-learnable). Registered as + # buffer so it moves with .cuda()/.to(). + self.register_buffer( + 'base_posenc', + _build_base_posenc(self.embed_dim, BASE_TOKEN_H, BASE_TOKEN_W), + persistent=False, + ) + + # Encoder. + self.encoder_layers = nn.ModuleList([ + _PreNormBlock(self.embed_dim, + cfg['enc_num_heads'], cfg['enc_head_dim'], + cfg['enc_mlp_hidden']) + for _ in range(cfg['enc_num_layers']) + ]) + self.encoder_final_norm = nn.LayerNorm(self.embed_dim) + + # Rnn core (stateful across T). + self.rnn_core = _GatedTransformerCore( + d_model=self.embed_dim, + num_heads=cfg['rnn_num_heads'], + head_dim=cfg['rnn_head_dim'], + mlp_hidden=cfg['rnn_mlp_hidden'], + num_xa_blocks=cfg['rnn_num_xa_blocks'], + ) + + # -- internals ------------------------------------------------------- + + def _tokenize(self, frames_bt: torch.Tensor) -> torch.Tensor: + """(B·T, 3, H, W) → (B·T, 1 + h·w, D) with posenc applied to patches + (CLS token is not posenc'd).""" + # Patchify. + x = self.patch_embed(frames_bt) # (B·T, D, h, w) + BT, D, h, w = x.shape + # Channel-last token layout. + x = x.permute(0, 2, 3, 1) # (B·T, h, w, D) + # Add posenc (resized from base 16×16 if needed). + posenc = _resize_posenc(self.base_posenc, h, w) # (h, w, D) + x = x + posenc + # Flatten spatial and prepend CLS. + x = x.reshape(BT, h * w, D) + cls = self.cls_token.to(x.dtype).expand(BT, 1, D) + return torch.cat([cls, x], dim=1) # (B·T, 1 + h·w, D) + + def _run_encoder(self, tokens: torch.Tensor) -> torch.Tensor: + for blk in self.encoder_layers: + tokens = blk(tokens) + return self.encoder_final_norm(tokens) + + # -- forward --------------------------------------------------------- + + def forward(self, video: torch.Tensor, + state: Optional[torch.Tensor] = None, + ) -> Tuple[torch.Tensor, torch.Tensor]: + """video: (B, T, 3, H, W) fp32 in [0, 1]. + + Returns: + features: (B, T, D, h, w) — CLS dropped, channel-first for + compatibility with our downstream pipeline. + final_state: (B, 1 + h·w, D) — carry across clip boundaries + at inference time. + """ + assert video.dim() == 5, f"expected (B,T,3,H,W); got {tuple(video.shape)}" + B, T, C, H, W = video.shape + assert C == 3 and H % PATCH_SIZE == 0 and W % PATCH_SIZE == 0, ( + f"H, W must be multiples of {PATCH_SIZE}; got {(H, W)}") + h = H // PATCH_SIZE + w = W // PATCH_SIZE + N = 1 + h * w + + # (1) Encoder — fully batched over B·T. + frames_bt = video.reshape(B * T, C, H, W) + tokens_bt = self._tokenize(frames_bt) # (B·T, N, D) + encoded_bt = self._run_encoder(tokens_bt) # (B·T, N, D) + encoded = encoded_bt.view(B, T, N, self.embed_dim) + + # (2) Rnn core — sequential over T. + if state is None: + state = encoded.new_zeros(B, N, self.embed_dim) + features = encoded.new_empty(B, T, N, self.embed_dim) + for t in range(T): + out, state = self.rnn_core(encoded[:, t], state) + features[:, t] = out + + # (3) Reshape: drop CLS, channel-first. + patch_feats = features[:, :, 1:, :] # (B, T, h·w, D) + patch_feats = patch_feats.view(B, T, h, w, self.embed_dim) + patch_feats = patch_feats.permute(0, 1, 4, 2, 3).contiguous() + return patch_feats, state + + +# --------------------------------------------------------------------------- +# Weight conversion: NPZ (JAX tree) → PyTorch state_dict +# --------------------------------------------------------------------------- + + +def _recover_tree(flat: dict) -> dict: + tree = {} + for k, v in flat.items(): + node = tree + parts = k.split('/') + for part in parts[:-1]: + node = node.setdefault(part, {}) + node[parts[-1]] = v + return tree + + +def _jax_dense_kernel_to_torch(k: np.ndarray) -> torch.Tensor: + """JAX Dense kernel (in_dim, out_dim) → PyTorch Linear weight (out_dim, in_dim).""" + return torch.from_numpy(np.ascontiguousarray(k.T.copy())).float() + + +def _jax_qkv_kernel_to_torch(k: np.ndarray) -> torch.Tensor: + """JAX DenseGeneral Q/K/V kernel (in_dim, num_heads, head_dim) + → PyTorch flat Linear weight (num_heads·head_dim, in_dim).""" + in_dim, H, D = k.shape + flat = k.reshape(in_dim, H * D) # (in_dim, inner) + return torch.from_numpy(np.ascontiguousarray(flat.T.copy())).float() + + +def _jax_qkv_bias_to_torch(b: np.ndarray) -> torch.Tensor: + """(num_heads, head_dim) → (num_heads·head_dim,)""" + return torch.from_numpy(np.ascontiguousarray(b.reshape(-1).copy())).float() + + +def _jax_out_kernel_to_torch(k: np.ndarray) -> torch.Tensor: + """JAX DenseGeneral output kernel (num_heads, head_dim, out_dim) + → PyTorch Linear weight (out_dim, num_heads·head_dim).""" + H, D, out_dim = k.shape + flat = k.reshape(H * D, out_dim) + return torch.from_numpy(np.ascontiguousarray(flat.T.copy())).float() + + +def _jax_out_bias_to_torch(b: np.ndarray) -> torch.Tensor: + return torch.from_numpy(np.ascontiguousarray(b.copy())).float() + + +def _convert_mha_block(sd_out: dict, prefix: str, src: dict): + """Fill sd_out for a `_MHA` module under `prefix.` from src subtree.""" + sd_out[prefix + 'query.weight'] = _jax_qkv_kernel_to_torch(src['query']['kernel']) + sd_out[prefix + 'query.bias'] = _jax_qkv_bias_to_torch(src['query']['bias']) + sd_out[prefix + 'key.weight'] = _jax_qkv_kernel_to_torch(src['key']['kernel']) + sd_out[prefix + 'key.bias'] = _jax_qkv_bias_to_torch(src['key']['bias']) + sd_out[prefix + 'value.weight'] = _jax_qkv_kernel_to_torch(src['value']['kernel']) + sd_out[prefix + 'value.bias'] = _jax_qkv_bias_to_torch(src['value']['bias']) + sd_out[prefix + 'out.weight'] = _jax_out_kernel_to_torch(src['out']['kernel']) + sd_out[prefix + 'out.bias'] = _jax_out_bias_to_torch(src['out']['bias']) + + +def _convert_ln(sd_out: dict, prefix: str, src: dict): + """JAX LayerNorm (`scale`, `bias`) → PyTorch (`weight`, `bias`).""" + sd_out[prefix + 'weight'] = torch.from_numpy(np.ascontiguousarray(src['scale'].copy())).float() + sd_out[prefix + 'bias'] = torch.from_numpy(np.ascontiguousarray(src['bias'].copy())).float() + + +def _convert_mlp(sd_out: dict, prefix: str, src: dict): + sd_out[prefix + 'dense_in.weight'] = _jax_dense_kernel_to_torch(src['dense_in']['kernel']) + sd_out[prefix + 'dense_in.bias'] = torch.from_numpy(np.ascontiguousarray(src['dense_in']['bias'].copy())).float() + sd_out[prefix + 'dense_out.weight'] = _jax_dense_kernel_to_torch(src['dense_out']['kernel']) + sd_out[prefix + 'dense_out.bias'] = torch.from_numpy(np.ascontiguousarray(src['dense_out']['bias'].copy())).float() + + +def _convert_gate_kernel(sd_out: dict, key: str, src_kernel: np.ndarray): + """GRU gate Linear(no bias). JAX kernel (D, D) → Linear weight (D, D) [transposed].""" + sd_out[key] = _jax_dense_kernel_to_torch(src_kernel) + + +def load_rvm_npz(model: RVMVideoSiamMAE, npz_path: str, + strict_load: bool = True, verbose: bool = True) -> None: + """Load the NPZ ckpt into `model` (in-place). + + Drops the MAE pretrain decoder, mask_token, and detokenizer paths — + only encoder + rnn_core + tokenizer + cls_token are consumed. + """ + raw = dict(np.load(npz_path, allow_pickle=False)) + tree = _recover_tree(raw) + + sd_out = {} + + # -- Tokenizer (patch embed). ------------------------------------------ + # JAX kernel shape (1, 16, 16, 3, D). T-kernel=1, so squeeze to (16, 16, 3, D) + # and permute to PyTorch Conv2d (D, 3, 16, 16). + conv_k = tree['tokenizer']['patch_embedding']['Conv_0']['kernel'] + conv_k = np.squeeze(conv_k, axis=0) # (16, 16, 3, D) + conv_k = np.transpose(conv_k, (3, 2, 0, 1)) # (D, 3, 16, 16) + sd_out['patch_embed.weight'] = torch.from_numpy(np.ascontiguousarray(conv_k.copy())).float() + conv_b = tree['tokenizer']['patch_embedding']['Conv_0']['bias'] + sd_out['patch_embed.bias'] = torch.from_numpy(np.ascontiguousarray(conv_b.copy())).float() + + # -- CLS token. -------------------------------------------------------- + sd_out['cls_token'] = torch.from_numpy(np.ascontiguousarray(tree['cls_token'].copy())).float() + + # -- Encoder layers. --------------------------------------------------- + enc = tree['encoder'] + for i in range(len(model.encoder_layers)): + src = enc[f'layers_{i}'] + prefix = f'encoder_layers.{i}.' + _convert_ln(sd_out, prefix + 'attention_norm.', src['attention_norm']) + _convert_mha_block(sd_out, prefix + 'attention.', src['attention']) + _convert_ln(sd_out, prefix + 'mlp_norm.', src['mlp_norm']) + _convert_mlp(sd_out, prefix + 'mlp.', src['mlp']) + + # Final encoder LayerNorm. In the npz it's named "LayerNorm_0". + _convert_ln(sd_out, 'encoder_final_norm.', enc['LayerNorm_0']) + + # -- Rnn core. --------------------------------------------------------- + rc = tree['rnn_core'] + _convert_gate_kernel(sd_out, 'rnn_core.input_update.weight', rc['input_update']['kernel']) + _convert_gate_kernel(sd_out, 'rnn_core.input_reset.weight', rc['input_reset']['kernel']) + _convert_gate_kernel(sd_out, 'rnn_core.state_update.weight', rc['state_update']['kernel']) + _convert_gate_kernel(sd_out, 'rnn_core.state_reset.weight', rc['state_reset']['kernel']) + # state_layer_norm — JAX has only `scale` (no bias). PyTorch LN has bias; + # we set weight=scale, bias=0 (and froze bias in __init__). + sd_out['rnn_core.state_layer_norm.weight'] = torch.from_numpy( + np.ascontiguousarray(rc['state_layer_norm']['scale'].copy())).float() + sd_out['rnn_core.state_layer_norm.bias'] = torch.zeros(model.embed_dim, dtype=torch.float32) + + # Transformer (xa_blocks + output_norm). + tfm = rc['transformer'] + for i in range(len(model.rnn_core.transformer.xa_blocks)): + src = tfm[f'xa_blocks_{i}'] + p = f'rnn_core.transformer.xa_blocks.{i}.' + _convert_ln(sd_out, p + 'ca_attention_norm.', src['ca_attention_norm']) + _convert_mha_block(sd_out, p + 'ca_attention.', src['ca_attention']) + _convert_ln(sd_out, p + 'mlp_norm.', src['mlp_norm']) + _convert_mlp(sd_out, p + 'mlp.', src['mlp']) + _convert_ln(sd_out, p + 'attention_norm.', src['attention_norm']) + _convert_mha_block(sd_out, p + 'attention.', src['attention']) + _convert_ln(sd_out, 'rnn_core.transformer.output_norm.', tfm['output_norm']) + + # -- Load. ------------------------------------------------------------- + missing, unexpected = model.load_state_dict(sd_out, strict=False) + if verbose: + print(f"[RVM] loaded {npz_path} ({len(sd_out)} tensors)") + # base_posenc is a non-persistent buffer — expected "missing" + missing_keys = [k for k in missing if k != 'base_posenc'] + if missing_keys: + print(f"[RVM] unexpected MISSING keys: {missing_keys[:8]}" + + (f" ... (+{len(missing_keys)-8} more)" if len(missing_keys) > 8 else "")) + if unexpected: + print(f"[RVM] UNEXPECTED keys in state_dict: {unexpected[:8]}" + + (f" ... (+{len(unexpected)-8} more)" if len(unexpected) > 8 else "")) + if strict_load and any(k != 'base_posenc' for k in missing): + raise RuntimeError(f"RVM ckpt load missing keys: {[k for k in missing if k != 'base_posenc']}") + + +# --------------------------------------------------------------------------- +# Convenience constructor +# --------------------------------------------------------------------------- + + +def build_rvm(variant: str = 'small', ckpt_path: Optional[str] = None, + device: str = 'cuda') -> RVMVideoSiamMAE: + """Build a frozen RVM model and optionally load weights from NPZ.""" + model = RVMVideoSiamMAE(variant=variant).to(device) + if ckpt_path is not None: + load_rvm_npz(model, ckpt_path) + # Freeze: RVM is a foundation model, not trained here. + for p in model.parameters(): + p.requires_grad_(False) + model.eval() + return model diff --git a/corridorkey/stage1.py b/corridorkey/stage1.py new file mode 100644 index 0000000000000000000000000000000000000000..a1998eeda085b3c8dca168e6930d3e9e33f33d3b --- /dev/null +++ b/corridorkey/stage1.py @@ -0,0 +1,920 @@ +"""Stage 1 — baton producer for CorridorKey v2. + +Pure-transformer spatiotemporal encoder/decoder on the /16 ↔ /64 pyramid: + + /1 RGB + hint + └ Patchify(k=16) → d=512 at /16 + └ 2 × FullSpatialBlock3D (encoder side; per-frame full H×W) + └ Downsample(k=8, s=4) → d=512 aux at /64 + └ cat frozen foundations + aux → d=1536 at /64 + └ first half of FullSpatioTemporalBlock stack + └ zero-init upsample to /16 + learned skip from encoder /16 + └ 4 × SwinBlock3D local /16 refinement + └ zero-init downsample add back to /64 + └ remaining FullSpatioTemporalBlock stack + └ Upsample4ICNR → d=512 at /16 + └ learned skip from latest /16 refinement + └ 8 × mixed Swin/CSwinBlock3D (decoder side) + with zero-init patch-stem skip before the final 2 blocks + → baton (d=512 at /16) + → aux head: Depatchify r=8 → 4-channel /2 output + initialized as half-res RGB passthrough when the 512d projected + RGB patch basis is available (pred_alpha = 0 init) + +No recurrence. Temporal mixing lives entirely in attention: global at /64 +(full spatiotemporal), windowed at /16 (Swin-3D with full-T windows and +balanced half-window spatial shift phases). + +Windows cleanly divide the /16 grid when `shape_quantum=128` (the +dataset contract for this model), so no intra-window padding is needed. +""" + +import torch +import torch.nn as nn +import torch.nn.functional as F + +from .blocks import ( + PatchifyConv, Downsample4, Upsample4ICNR, Depatchify, + FullSpatialBlock3D, SwinBlock3D, CSwinBlock3D, + FullSpatioTemporalBlock, FullResolutionRecurrentMemorySite, + FusionLinSwiglu, + init_orthogonal_rgb_passthrough_, +) +from .rope import RoPE2D, RoPE3D + + +@torch.no_grad() +def _init_center_area_downsample_(conv: nn.Conv2d) -> None: + """Initialize k=8/s=4 downsample as exact per-cell area pooling. + + The central stride×stride region averages the corresponding input cell; + all spill taps start at zero. When channel counts differ, a semi-orthogonal + channel projection is used before the same spatial average. + """ + out_ch, in_ch, kh, kw = conv.weight.shape + sy, sx = conv.stride + if kh < sy or kw < sx: + raise ValueError( + f"downsample kernel {(kh, kw)} must cover stride {(sy, sx)}") + y0 = (kh - sy) // 2 + x0 = (kw - sx) // 2 + if (kh - sy) % 2 != 0 or (kw - sx) % 2 != 0: + raise ValueError( + f"downsample kernel-stride delta must be even; " + f"kernel={(kh, kw)} stride={(sy, sx)}") + + conv.weight.zero_() + if conv.bias is not None: + conv.bias.zero_() + + if out_ch == in_ch: + proj = torch.eye(out_ch, device=conv.weight.device, dtype=conv.weight.dtype) + elif out_ch < in_ch: + q, r = torch.linalg.qr( + torch.randn(in_ch, out_ch, device=conv.weight.device, + dtype=torch.float32), + mode='reduced', + ) + signs = torch.sign(torch.diagonal(r)) + signs = torch.where(signs == 0, torch.ones_like(signs), signs) + proj = (q * signs.view(1, -1)).T.to(dtype=conv.weight.dtype) + else: + q, r = torch.linalg.qr( + torch.randn(out_ch, in_ch, device=conv.weight.device, + dtype=torch.float32), + mode='reduced', + ) + signs = torch.sign(torch.diagonal(r)) + signs = torch.where(signs == 0, torch.ones_like(signs), signs) + proj = (q * signs.view(1, -1)).to(dtype=conv.weight.dtype) + + conv.weight[:, :, y0:y0 + sy, x0:x0 + sx].copy_( + proj[:, :, None, None] / float(sy * sx)) + + +def _frontloaded_qkv_kernels(total: int, kernel: int, + fraction: float = 1.0) -> list[int]: + kernel = int(kernel or 0) + total = int(total) + if total <= 0 or kernel <= 0: + return [0 for _ in range(max(total, 0))] + frac = max(0.0, min(float(fraction), 1.0)) + active = total if frac >= 1.0 else int(round(total * frac)) + active = max(0, min(total, active)) + return [kernel if i < active else 0 for i in range(total)] + + +def _stage1_qkv_kernels(n_enc: int, n_body: int, n_mid: int, n_dec: int, + kernel: int, fraction: float = 1.0) -> list[int]: + plan = _frontloaded_qkv_kernels( + n_enc + n_body + n_mid + n_dec, + kernel, + fraction, + ) + if not plan or int(kernel or 0) <= 0 or float(fraction) >= 1.0: + return plan + + # The front-loaded plan would spend the first /16 slots on enc_16, which + # is now full-spatial attention. Move those /16 QKV-DW slots onto the + # local mid_16 Swin detour instead; keep the /64 body allocation unchanged. + enc_active = [i for i in range(n_enc) if plan[i] > 0] + if not enc_active or n_mid <= 0: + return plan + + for i in enc_active: + plan[i] = 0 + + mid_start = n_enc + n_body + mid_slots = [ + mid_start + i for i in range(n_mid) + if plan[mid_start + i] == 0 + ] + placed = 0 + for i in mid_slots[:len(enc_active)]: + plan[i] = int(kernel) + placed += 1 + + # Preserve the requested active count if a nonstandard layout has fewer + # empty mid slots than relocated enc slots. + for i in enc_active[placed:]: + plan[i] = int(kernel) + return plan + + +def _qkv_kernel_at(kernels, i: int) -> int: + if isinstance(kernels, (list, tuple)): + return int(kernels[i]) if i < len(kernels) else 0 + return int(kernels or 0) + + +def _stage1_dec_blocks(d_16, num_heads_16, window_HW, mlp_mult, rope_16, + n_blocks=8, local_mix_kernel=0, + local_mix_init_std=None, mlp_dw_kernel=0, + qkv_dw_kernel=0): + """Build the decoder pattern. + + An earlier layout used 6 blocks: 2 dense Swin → 2 CSwin → 2 dense Swin. + The current layout uses 8 blocks: 2 dense Swin → CSwin → 2 inserted dense Swin + → CSwin → 2 dense Swin. The CSwin pair still gives two + full-global sweeps (each block mixes both H and W axes in parallel + heads), with a small local refinement stack between the two sweeps. + + Dense Swin shifts expose all four half-window phases. The final pair is + the orthogonal H-only/W-only pair because the late patch-stem skip enters + immediately before those blocks. + """ + w = window_HW + hw = w // 2 + if n_blocks == 6: + specs = [ + ("swin", (0, 0)), + ("swin", (hw, hw)), + ("cswin", None), + ("cswin", None), + ("swin", (hw, 0)), + ("swin", (0, hw)), + ] + elif n_blocks == 8: + specs = [ + ("swin", (0, 0)), + ("swin", (hw, hw)), + ("cswin", None), + ("swin", (hw, 0)), + ("swin", (0, hw)), + ("cswin", None), + ("swin", (hw, 0)), + ("swin", (0, hw)), + ] + else: + raise ValueError( + f"n_dec_16 must be 6 or 8 for the fixed CSwin pattern; got {n_blocks}") + blocks = [] + for i, (kind, shift) in enumerate(specs): + k = _qkv_kernel_at(qkv_dw_kernel, i) + if kind == "swin": + blocks.append( + SwinBlock3D(d_16, num_heads_16, w, w, shift=shift, + mlp_mult=mlp_mult, rope=rope_16, + local_mix_kernel=local_mix_kernel, + local_mix_init_std=local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=k) + ) + else: + blocks.append( + CSwinBlock3D(d_16, num_heads_16, + mlp_mult=mlp_mult, rope=rope_16, + local_mix_kernel=local_mix_kernel, + local_mix_init_std=local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=k) + ) + return blocks + + +def _stage1_mid16_blocks(d_16, num_heads_16, window_HW, mlp_mult, rope_16, + n_blocks=2, local_mix_kernel=0, + local_mix_init_std=None, mlp_dw_kernel=0, + qkv_dw_kernel=0): + w = window_HW + hw = w // 2 + phases = [(0, 0), (hw, hw), (hw, 0), (0, hw)] + return [ + SwinBlock3D(d_16, num_heads_16, w, w, + shift=phases[i % len(phases)], + mlp_mult=mlp_mult, rope=rope_16, + local_mix_kernel=local_mix_kernel, + local_mix_init_std=local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=_qkv_kernel_at(qkv_dw_kernel, i)) + for i in range(int(n_blocks)) + ] + + +class Stage1(nn.Module): + def __init__( + self, + # Channel widths + d_16: int = 512, + d_64: int = 1536, + cradio_dim: int = 0, + moge_dim: int = 768, + rvm_dim: int = 0, # 0 disables RVM feature fusion + # Depth + n_enc_16: int = 2, + n_body_64: int = 10, + n_dec_16: int = 6, + # Heads — d_16=512 with 8 heads gives head_dim=64; RoPE3D splits + # pairs nearly evenly across T/H/W. + num_heads_16: int = 8, # 512 / 8 = 64 + num_heads_64: int = 16, # 1536 / 16 = 96 + # Windowing (spatial only — temporal is always full). Window=16 at + # /16 means each Swin-3D block attends across 16×16 spatial tokens + # (256 pixel receptive field per block). The V+H axial pair in the + # middle of dec_16 gives full global mixing, so the windowed blocks + # only need local refinement at a smaller window. SwinAttention3D + # pads internally for /16 grids not divisible by 16 (shape_quantum=128 + # guarantees /8 divisibility at /16 — w=16 may still need padding + # for oddly-sized crops). + window_HW: int = 16, + local_mix_kernel: int = 7, + local_mix_init_std: float | None = None, + body64_local_mix_kernel: int = 7, + body64_local_mix_init_std: float | None = None, + body64_mlp_hidden: int | None = None, + body64_direct_concat: bool = False, + recurrent_64_sites: tuple[int, ...] | None = None, + hint_adaln_blocks: int = 2, + mid_16_blocks: int = 4, + mlp_dw_kernel: int = 0, + qkv_dw_kernel: int = 0, + qkv_dw_fraction: float = 1.0, + # MLP + mlp_mult: float = 2.5, + # Per-block gradient checkpointing during training. Each Swin-3D / + # Full-ST block is individually checkpointed → peak activation memory + # during backward is ~one block's activations, not the whole stage. + gradient_checkpoint: bool = False, + # Overlap patchify/depatchify (stem + aux_depatchify). + # overlap_patchify is a legacy bool alias (True → mult=3). + # overlap_mult picks the receptive-field multiplier: + # 1 = no overlap; 2 = half-overlap (kernel=2·patch); + # 3 = full-overlap (kernel=3·patch, immediate + 8 neighbors). + overlap_patchify: bool = False, + overlap_mult: int = 2, + depatch_fourier_features: int = 0, + depatch_fourier_kernel: int = 3, + # MAE-style pretraining hooks. Disabled in the normal inference path. + enable_mae_mask_tokens: bool = False, + bottleneck_head_stride: int | None = 8, + # RoPE training-time augmentations (DINOv3-style, per-forward coherent). + # None / 0 / 1.0 respectively disables each. + rope_shift_coords: float = None, + rope_jitter_coords: float = None, + rope_rescale_coords: float = None, + rope_base: float = 100.0, + ): + super().__init__() + self.d_16 = d_16 + self.d_64 = d_64 + self.window_HW = window_HW + self.gradient_checkpoint = gradient_checkpoint + self.enable_mae_mask_tokens = bool(enable_mae_mask_tokens) + self.bottleneck_head_stride = bottleneck_head_stride + self.hint_adaln_blocks = max(0, int(hint_adaln_blocks)) + self.mid_16_blocks_count = max(0, int(mid_16_blocks)) + self.recurrent_64_sites = tuple( + int(i) for i in (recurrent_64_sites or ()) + ) + if len(set(self.recurrent_64_sites)) != len(self.recurrent_64_sites): + raise ValueError( + f"recurrent_64_sites contains duplicates: " + f"{self.recurrent_64_sites}") + bad_sites = [ + i for i in self.recurrent_64_sites + if i < 0 or i >= int(n_body_64) + ] + if bad_sites: + raise ValueError( + f"recurrent_64_sites {bad_sites} outside body block range " + f"0..{int(n_body_64) - 1}") + qkv_plan = _stage1_qkv_kernels( + n_enc_16, + n_body_64, + self.mid_16_blocks_count, + n_dec_16, + qkv_dw_kernel, + qkv_dw_fraction, + ) + qkv_pos = 0 + qkv_enc = qkv_plan[qkv_pos:qkv_pos + n_enc_16] + qkv_pos += n_enc_16 + qkv_body = qkv_plan[qkv_pos:qkv_pos + n_body_64] + qkv_pos += n_body_64 + qkv_mid = qkv_plan[qkv_pos:qkv_pos + self.mid_16_blocks_count] + qkv_pos += self.mid_16_blocks_count + qkv_dec = qkv_plan[qkv_pos:qkv_pos + n_dec_16] + self.qkv_dw_active_blocks = sum(1 for k in qkv_plan if k > 0) + self.qkv_dw_total_blocks = len(qkv_plan) + + # Stem: 4 input channels (RGB + hint) → d_16 at /16 via k=16 s=16. + # Normal Kaiming init (no zero-init here — we want the stem to + # actually produce features from t=0 since stage 1 has nothing + # else at /16 to take up the slack). + self.stem = PatchifyConv(4, d_16, patch=16, zero_init_first=0, + overlap=overlap_patchify, + overlap_mult=overlap_mult) + + # Shared RoPE instances per attention scale — saves frequency tables + # and (negligibly) memory. The pre-down /16 blocks use spatial-only + # RoPE; later /16 and /64 blocks use spatiotemporal RoPE. + head_dim_16 = d_16 // num_heads_16 + head_dim_64 = d_64 // num_heads_64 + _rope_kw = dict(base=rope_base, + shift_coords=rope_shift_coords, + jitter_coords=rope_jitter_coords, + rescale_coords=rope_rescale_coords) + self.rope_16_spatial = RoPE2D(head_dim_16, **_rope_kw) + self.rope_16 = RoPE3D(head_dim_16, **_rope_kw) + self.rope_64 = RoPE3D(head_dim_64, **_rope_kw) + + # /16 encoder side: per-frame full-spatial attention. This gives the + # high-res hint/RGB features global spatial context before downsample, + # while temporal mixing remains concentrated in the /64 body. + assert n_enc_16 >= 1 + self.enc_16 = nn.ModuleList([ + FullSpatialBlock3D( + d_16, num_heads_16, + mlp_mult=mlp_mult, rope=self.rope_16_spatial, + local_mix_kernel=local_mix_kernel, + local_mix_init_std=local_mix_init_std, + adaln_cond_dim=(1 if i < self.hint_adaln_blocks else 0), + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=qkv_enc[i], + ) + for i in range(n_enc_16) + ]) + + # /16 → /64 downsample — kernel-8 stride-4. Produces d_down_aux=d_16 + # channels that get concat'd with VFM → FusionLinSwiglu + # → d_64. Init as exact center-cell area pooling so the aux stream has + # useful /16-derived content immediately; fusion still zeros aux + # contribution at t=0, so the model function remains VFM-only initially. + d_down_aux = d_16 # 512 by default + self.d_down_aux = d_down_aux + self.cradio_dim = cradio_dim + self.moge_dim = moge_dim + self.rvm_dim = rvm_dim + self.vfm_channels = cradio_dim + moge_dim + rvm_dim + self.down = Downsample4(d_16, d_down_aux, kernel=8, stride=4, norm=False) + _init_center_area_downsample_(self.down.conv) + + # Fusion block: linear projection with SwiGLU wrapped AROUND it. + # in_dim = vfm + d_down_aux + # out_dim = d_64 (1536) + # hidden = in_dim · mlp_mult + # Linear shortcut: semi-orthogonal VFM projection, zero-for-aux. + # SwiGLU: up-projects from full in_dim before down-project, so the + # nonlinear gate sees the full input before compression. w_down is + # zero-inited → SwiGLU contributes 0 at t=0. + # Hidden size is pinned before RVM so retrofitting the recurrent + # stream does not expand an already-trained fusion MLP. C-RADIO is + # not a retrofit path for existing checkpoints; when explicitly enabled, + # it is treated as part of the base frozen-foundation budget. + pre_rvm_in = ( + self.cradio_dim + self.moge_dim + d_down_aux + ) + hidden_pinned = int(round(pre_rvm_in * mlp_mult)) + self.body64_direct_concat = bool(body64_direct_concat) + fuse_in_dim = self.vfm_channels + d_down_aux + if self.body64_direct_concat: + if d_64 != fuse_in_dim: + raise ValueError( + f"body64_direct_concat requires d_64 == VFM+aux " + f"({fuse_in_dim}); got d_64={d_64}") + self.fuse = None + else: + self.fuse = FusionLinSwiglu( + vfm_dim=self.vfm_channels, aux_dim=d_down_aux, + out_dim=d_64, mult=mlp_mult, bias=True, + hidden=hidden_pinned, + ) + + # /64 body. Full spatiotemporal attention — no windowing. Token + # count (H/64 · W/64 · T) is small enough that full quadratic + # attention is cheap here. + self.body_64 = nn.ModuleList([ + FullSpatioTemporalBlock(d_64, num_heads_64, mlp_mult=mlp_mult, + rope=self.rope_64, + local_mix_kernel=body64_local_mix_kernel, + local_mix_init_std=body64_local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + mlp_hidden=body64_mlp_hidden, + qkv_dw_kernel=qkv_body[i]) + for i in range(n_body_64) + ]) + self.body_64_split = n_body_64 // 2 + self.recurrent_64 = nn.ModuleList([ + FullResolutionRecurrentMemorySite( + d_64, num_heads_64, rope=RoPE2D(head_dim_64, **_rope_kw)) + for _ in self.recurrent_64_sites + ]) + self._recurrent_64_index = { + site: i for i, site in enumerate(self.recurrent_64_sites) + } + + # Mid-stage /64 → /16 → /64 detour. The up/down projections are + # zero-inited so the loop is exact-identity at init while still + # allowing a local /16 refinement route to grow during training. + self.mid_up = Upsample4ICNR(d_64, d_16, r=4, post_smooth=True, norm=False) + self.mid_skip_scale = nn.Parameter(torch.ones(d_16)) + self.mid_16 = nn.ModuleList( + _stage1_mid16_blocks( + d_16, num_heads_16, window_HW, mlp_mult, self.rope_16, + n_blocks=self.mid_16_blocks_count, + local_mix_kernel=local_mix_kernel, + local_mix_init_std=local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=qkv_mid, + ) + ) + self.mid_down = Downsample4(d_16, d_64, kernel=8, stride=4, norm=False) + with torch.no_grad(): + nn.init.zeros_(self.mid_up.proj.weight) + if self.mid_up.proj.bias is not None: + nn.init.zeros_(self.mid_up.proj.bias) + nn.init.zeros_(self.mid_down.conv.weight) + if self.mid_down.conv.bias is not None: + nn.init.zeros_(self.mid_down.conv.bias) + + # /64 → /16 upsample. ICNR-inited + post-conv smoothing to mimic + # the overlap the kernel-8 downsample provided on the encoder side. + self.up = Upsample4ICNR(d_64, d_16, r=4, post_smooth=True, norm=False) + + # Learned per-channel U-Net skip from /16 encoder → /16 decoder. + # Shape (d_16,). Initialized to 1.0 — at t=0 the decoder sees the + # encoder /16 features at full strength. The model can learn to + # scale this down if the /64 body's output is preferred. Alternative + # was init=0 (start with no skip, grow), but encoder features contain + # early-pipeline RGB-derived detail that's useful to have available + # from step 1. + self.uskip_scale = nn.Parameter(torch.ones(d_16)) + self.patch_late_skip_scale = nn.Parameter(torch.zeros(d_16)) + + # /16 decoder side: mixed Swin/CSwin pattern. The 6-block default is + # the compact 2 Swin → 2 CSwin → 2 Swin layout; the optional 8-block + # layout inserts a local Swin pair between the two CSwin sweeps. The + # late patch-stem skip is injected before the final Swin pair. + assert n_dec_16 in (6, 8), ( + f"n_dec_16 must be 6 or 8 for the CSwin-in-the-middle pattern; got {n_dec_16}.") + self.dec_16 = nn.ModuleList( + _stage1_dec_blocks(d_16, num_heads_16, window_HW, mlp_mult, + self.rope_16, n_blocks=n_dec_16, + local_mix_kernel=local_mix_kernel, + local_mix_init_std=local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=qkv_dec) + ) + + # Aux head: baton at /16 (d=512) → 4-channel prediction at /2 via + # 8× depatchify. With production dims and non-overlap patching, the + # FG rows are initialized as area-downsampled inverse RGB patch basis + # and alpha rows remain zero. + self.aux_depatchify = Depatchify(d_16, out_channels=4, r=8, + zero_init_all=True, bias=True, + overlap=overlap_patchify, + overlap_mult=overlap_mult, + fourier_features=depatch_fourier_features, + fourier_kernel=depatch_fourier_kernel) + self.rgb_passthrough_init = init_orthogonal_rgb_passthrough_( + self.stem, self.aux_depatchify, + token_offset=0, + output_rgb_offset=1, + downsample=2, + ) + if self.rgb_passthrough_init: + nn.init.zeros_(self.up.proj.weight) + if self.up.proj.bias is not None: + nn.init.zeros_(self.up.proj.bias) + if self.enable_mae_mask_tokens: + self.mae_token_16 = nn.Parameter(torch.zeros(d_16)) + self.mae_token_64_aux = nn.Parameter(torch.zeros(d_down_aux)) + if self.vfm_channels > 0: + self.mae_token_vfm = nn.Parameter(torch.zeros(self.vfm_channels)) + if bottleneck_head_stride is not None: + if bottleneck_head_stride not in (4, 8): + raise ValueError( + "bottleneck_head_stride must be 4 or 8 when enabled") + r = 64 // int(bottleneck_head_stride) + self.bottleneck_depatchify = Depatchify( + d_64, out_channels=4, r=r, + zero_init_all=True, bias=True, + overlap=overlap_patchify, + overlap_mult=overlap_mult, + fourier_features=depatch_fourier_features, + fourier_kernel=depatch_fourier_kernel, + ) + + # ----------------------------------------------------------------- + + def _ckpt_block( + self, + blk, + x: torch.Tensor, + t_stride: float, + adaln_cond: torch.Tensor | None = None, + ) -> torch.Tensor: + """Call `blk(x, t_stride=t_stride)` — wrapped in `torch.utils.checkpoint` + when training with gradient_checkpoint enabled so only one block's + activations are resident at a time during backward.""" + if adaln_cond is not None: + if self.gradient_checkpoint and self.training and torch.is_grad_enabled(): + return torch.utils.checkpoint.checkpoint( + blk, x, t_stride, adaln_cond, use_reentrant=False, + ) + return blk(x, t_stride, adaln_cond) + if self.gradient_checkpoint and self.training and torch.is_grad_enabled(): + return torch.utils.checkpoint.checkpoint( + blk, x, t_stride, use_reentrant=False, + ) + return blk(x, t_stride=t_stride) + + def _swin3d_run( + self, + blocks, + x_5d: torch.Tensor, + t_stride: float, + hint_quality: torch.Tensor | None = None, + late_skip_5d: torch.Tensor | None = None, + late_skip_scale: torch.Tensor | None = None, + late_skip_index: int | None = None, + ) -> torch.Tensor: + """Run a list of Swin3D blocks on a 5D tensor (B, T, C, H, W). + + Swin blocks accept (B, T, H, W, C), so permute in & out here. + """ + frame_chunk = int(getattr(self, "low_vram_enc_frame_chunk", 0) or 0) + if (frame_chunk > 0 and not self.training and late_skip_5d is None + and len(blocks) > 0 + and all(isinstance(blk, FullSpatialBlock3D) for blk in blocks) + and x_5d.shape[1] > frame_chunk): + outs = [] + for t0 in range(0, x_5d.shape[1], frame_chunk): + t1 = min(t0 + frame_chunk, x_5d.shape[1]) + x = x_5d[:, t0:t1].permute(0, 1, 3, 4, 2).contiguous() + hq = ( + hint_quality[:, t0:t1] + if hint_quality is not None else None + ) + for blk in blocks: + cond = hq if getattr(blk, "adaln", None) is not None else None + x = self._ckpt_block(blk, x, t_stride, adaln_cond=cond) + outs.append(x.permute(0, 1, 4, 2, 3).contiguous()) + return torch.cat(outs, dim=1) + + # (B, T, C, H, W) → (B, T, H, W, C) + x = x_5d.permute(0, 1, 3, 4, 2).contiguous() + skip = None + if late_skip_5d is not None: + skip = late_skip_5d.permute(0, 1, 3, 4, 2).contiguous() + if late_skip_scale is None: + raise ValueError("late_skip_scale is required with late_skip_5d") + if late_skip_index is None: + late_skip_index = max(len(blocks) - 2, 0) + scale = late_skip_scale.to(device=x.device, dtype=x.dtype).view( + 1, 1, 1, 1, -1) + else: + scale = None + for i, blk in enumerate(blocks): + if skip is not None and i == late_skip_index: + x = x + skip * scale + cond = hint_quality if getattr(blk, "adaln", None) is not None else None + x = self._ckpt_block(blk, x, t_stride, adaln_cond=cond) + return x.permute(0, 1, 4, 2, 3).contiguous() + + def _full_run(self, blocks, x_5d: torch.Tensor, t_stride: float) -> torch.Tensor: + """Run full-spatiotemporal blocks on (B, T, C, H, W).""" + x = x_5d.permute(0, 1, 3, 4, 2).contiguous() + for blk in blocks: + x = self._ckpt_block(blk, x, t_stride) + return x.permute(0, 1, 4, 2, 3).contiguous() + + def _full_run_nthwc( + self, + blocks, + x: torch.Tensor, + t_stride: float, + ) -> torch.Tensor: + """Run full-spatiotemporal blocks on native (B, T, H, W, C) layout.""" + for blk in blocks: + x = self._ckpt_block(blk, x, t_stride) + return x + + def _ckpt_body64_with_recurrent( + self, + blk, + site, + x: torch.Tensor, + state: torch.Tensor | None, + t_stride: float, + ) -> tuple[torch.Tensor, torch.Tensor]: + if self.gradient_checkpoint and self.training and torch.is_grad_enabled(): + if state is None: + def fn(inp): + return blk( + inp, t_stride, + recurrent_site=site, + recurrent_state=None, + ) + return torch.utils.checkpoint.checkpoint( + fn, x, use_reentrant=False, + ) + + def fn(inp, st): + return blk( + inp, t_stride, + recurrent_site=site, + recurrent_state=st, + ) + return torch.utils.checkpoint.checkpoint( + fn, x, state, use_reentrant=False, + ) + return blk( + x, t_stride=t_stride, + recurrent_site=site, + recurrent_state=state, + ) + + def _run_body64_range( + self, + start: int, + end: int, + x: torch.Tensor, + t_stride: float, + recurrent_states: list[torch.Tensor | None] | None, + ) -> tuple[torch.Tensor, list[torch.Tensor | None] | None]: + for block_idx in range(start, end): + site_idx = self._recurrent_64_index.get(block_idx) + if site_idx is not None: + if recurrent_states is None: + raise RuntimeError( + "internal error: recurrent site active without states") + x, recurrent_states[site_idx] = self._ckpt_body64_with_recurrent( + self.body_64[block_idx], + self.recurrent_64[site_idx], + x, + recurrent_states[site_idx], + t_stride, + ) + else: + x = self._ckpt_block(self.body_64[block_idx], x, t_stride) + return x, recurrent_states + + def _apply_mae_token( + self, + x: torch.Tensor, + mask: torch.Tensor | None, + token: torch.Tensor, + name: str, + ) -> torch.Tensor: + if mask is None: + return x + if not self.enable_mae_mask_tokens: + raise RuntimeError(f"{name} received an MAE mask but MAE tokens are disabled") + expected = (x.shape[0], x.shape[1], 1, x.shape[-2], x.shape[-1]) + if tuple(mask.shape) != expected: + raise ValueError( + f"{name} mask shape {tuple(mask.shape)} != expected {expected}") + mask = mask.to(device=x.device, dtype=torch.bool) + tok = token.to(device=x.device, dtype=x.dtype).view(1, 1, -1, 1, 1) + return torch.where(mask, tok, x) + + @staticmethod + def _resolve_hint_quality( + hint: torch.Tensor, + hint_quality: torch.Tensor | None, + ) -> torch.Tensor: + if hint_quality is None: + return (hint.detach().amax(dim=(-1, -2, -3)) > 1e-5).to(dtype=hint.dtype) + q = hint_quality.to(device=hint.device, dtype=hint.dtype) + while q.dim() > 2 and q.shape[-1] == 1: + q = q.squeeze(-1) + if q.dim() != 2: + raise ValueError( + f"hint_quality must be (B,T) or trailing-singleton equivalent; got {tuple(q.shape)}") + if tuple(q.shape) != tuple(hint.shape[:2]): + raise ValueError( + f"hint_quality shape {tuple(q.shape)} != hint B,T {tuple(hint.shape[:2])}") + return q.clamp_(-1.0, 1.0) + + def forward( + self, + rgb: torch.Tensor, # (B, T, 3, H, W) signed asinh encoded scene-linear RGB + hint: torch.Tensor, # (B, T, 1, H, W) in [0, 1] + moge_feat: torch.Tensor, # (B, T, moge_dim, H/64, W/64) + cradio_feat: 'torch.Tensor | None' = None, # (B, T, cradio_dim, H/64, W/64) + rvm_feat: 'torch.Tensor | None' = None, # (B, T, rvm_dim, H/64, W/64) if enabled + t_stride: float = 1.0, + mae_mask_16: 'torch.Tensor | None' = None, # (B,T,1,H/16,W/16) + mae_mask_64: 'torch.Tensor | None' = None, # (B,T,1,H/64,W/64) + mae_vfm_mask_64: 'torch.Tensor | None' = None, # (B,T,1,H/64,W/64) + hint_quality: 'torch.Tensor | None' = None, # (B,T), -1=decide subject, 0=no frame hint, 1=perfect + recurrent_states: 'tuple[torch.Tensor | None, ...] | None' = None, + ) -> dict: + """Returns dict with: + baton: (B, T, d_16, H/16, W/16) — feeds stage 2 + aux: (B, T, 4, H/2, W/2) — stage-1 supervised output + """ + B, T, _, H, W = rgb.shape + assert H % 64 == 0 and W % 64 == 0, ( + f"Stage1: H,W must be multiples of 64 (shape_quantum); got {(H, W)}") + + # Merge B·T for conv ops that don't care about T + BT = B * T + hint_quality = self._resolve_hint_quality(hint, hint_quality) + x_in = torch.cat([rgb, hint], dim=2).reshape(BT, 4, H, W) + + # /1 → /16 patchify + f_16 = self.stem(x_in) # (BT, d_16, H/16, W/16) + del x_in + f_16 = f_16.reshape(B, T, self.d_16, H // 16, W // 16) + if mae_mask_16 is not None: + f_16 = self._apply_mae_token(f_16, mae_mask_16, + self.mae_token_16, "stage1 /16") + f_16_patch = f_16 + + # /16 encoder side (per-frame full-spatial). Keep a handle for the + # U-skip before the downsample eats the spatial detail. + f_16_enc = self._swin3d_run( + self.enc_16, f_16, t_stride, hint_quality=hint_quality) + + # /16 → /64 downsample (center-cell area init; produces d_down_aux channels) + f_16_flat = f_16_enc.reshape(BT, self.d_16, H // 16, W // 16) + f_64_down = self.down(f_16_flat) # (BT, d_down_aux, H/64, W/64) + del f_16_flat + h_64, w_64 = H // 64, W // 64 + f_64_down = f_64_down.reshape(B, T, self.d_down_aux, h_64, w_64) + if mae_mask_64 is not None: + f_64_down = self._apply_mae_token( + f_64_down, mae_mask_64, self.mae_token_64_aux, + "stage1 /64 aux") + + # VFM features at /64 (already channel-normalized by FrozenFoundations). + # Concat order is fixed at [cradio | moge | rvm] — the fuse layer's + # linear_shortcut init relies on this ordering. + assert moge_feat.shape[-2:] == (h_64, w_64), ( + f"moge_feat spatial {moge_feat.shape[-2:]} != /64 ({(h_64, w_64)})") + vfm_parts = [] + if self.cradio_dim > 0: + assert cradio_feat is not None, ( + "cradio_dim > 0 requires cradio_feat to be passed in") + assert cradio_feat.shape[-2:] == (h_64, w_64), ( + f"cradio_feat spatial {cradio_feat.shape[-2:]} != /64 ({(h_64, w_64)})") + vfm_parts.append(cradio_feat) + vfm_parts.append(moge_feat) + if self.rvm_dim > 0: + assert rvm_feat is not None, ( + "rvm_dim > 0 requires rvm_feat to be passed in") + # RVM shares C-RADIO's input (H/4) and patch size (16), so features + # land on the /64 grid directly — no resize needed. + assert rvm_feat.shape[-2:] == (h_64, w_64), ( + f"rvm_feat spatial {rvm_feat.shape[-2:]} != /64 ({(h_64, w_64)}); " + f"RVM should be fed at C-RADIO's H/4 input size") + vfm_parts.append(rvm_feat) + vfm_cat = torch.cat(vfm_parts, dim=2) # (B, T, vfm_C, h, w) + + # Pad VFM to match self.vfm_channels if something got disabled + # (e.g. one foundation off in smoke-test) so the SwiGLU input shape + # stays constant. + pad_c = self.vfm_channels - vfm_cat.shape[2] + if pad_c > 0: + vfm_cat = F.pad(vfm_cat, (0, 0, 0, 0, 0, pad_c)) + if mae_vfm_mask_64 is not None: + if not hasattr(self, "mae_token_vfm"): + raise RuntimeError( + "received an MAE VFM mask but VFM MAE token is disabled") + vfm_cat = self._apply_mae_token( + vfm_cat, mae_vfm_mask_64, self.mae_token_vfm, + "stage1 /64 VFM") + + # Full concat at /64 (vfm + downsampled aux) → either direct /64 body + # input or fusion (linear+SwiGLU). + combined = torch.cat([vfm_cat, f_64_down], dim=2) # (B, T, vfm+aux, h, w) + del vfm_cat, f_64_down, vfm_parts + tokens = combined.permute(0, 1, 3, 4, 2).contiguous() # (B, T, h, w, fuse_in_dim) + del combined + f_64 = tokens if self.fuse is None else self.fuse(tokens) # (B, T, h, w, d_64) + del tokens + + # /64 body, with a mid-stage detour back through /16 for local + # refinement before returning to /64 for the remaining global blocks. + recurrent_states_out = None + if len(self.recurrent_64) > 0: + if recurrent_states is None: + recurrent_states_out = [None for _ in self.recurrent_64] + else: + if len(recurrent_states) != len(self.recurrent_64): + raise ValueError( + f"got {len(recurrent_states)} recurrent states for " + f"{len(self.recurrent_64)} recurrent sites") + recurrent_states_out = list(recurrent_states) + + split = self.body_64_split + if recurrent_states_out is None: + f_64 = self._full_run_nthwc(self.body_64[:split], f_64, t_stride) + else: + f_64, recurrent_states_out = self._run_body64_range( + 0, split, f_64, t_stride, recurrent_states_out) + + f_64_chw = f_64.permute(0, 1, 4, 2, 3).contiguous() + f_64_mid_flat = f_64_chw.reshape(BT, self.d_64, H // 64, W // 64) + del f_64_chw + f_16_mid_up = self.mid_up(f_64_mid_flat).reshape( + B, T, self.d_16, H // 16, W // 16) + del f_64_mid_flat + mid_scale = self.mid_skip_scale.view(1, 1, self.d_16, 1, 1) + f_16_mid = f_16_mid_up + mid_scale * f_16_enc + del f_16_mid_up, f_16_enc + if len(self.mid_16) > 0: + f_16_mid = self._swin3d_run(self.mid_16, f_16_mid, t_stride) + f_16_mid_flat = f_16_mid.reshape(BT, self.d_16, H // 16, W // 16) + f_64_delta = self.mid_down(f_16_mid_flat).reshape( + B, T, self.d_64, H // 64, W // 64) + del f_16_mid_flat + f_64 = f_64 + f_64_delta.permute(0, 1, 3, 4, 2).contiguous() + del f_64_delta + + if recurrent_states_out is None: + f_64 = self._full_run_nthwc(self.body_64[split:], f_64, t_stride) + else: + f_64, recurrent_states_out = self._run_body64_range( + split, len(self.body_64), f_64, t_stride, + recurrent_states_out) + f_64_chw = f_64.permute(0, 1, 4, 2, 3).contiguous() + del f_64 + + bottleneck_out = None + if self.bottleneck_head_stride is not None: + f_64_head = f_64_chw.reshape(BT, self.d_64, H // 64, W // 64) + raw = self.bottleneck_depatchify(f_64_head) + del f_64_head + s = int(self.bottleneck_head_stride) + bottleneck_out = raw.reshape(B, T, 4, H // s, W // s) + del raw + + # /64 → /16 upsample + f_64_flat = f_64_chw.reshape(BT, self.d_64, H // 64, W // 64) + del f_64_chw + f_16_up = self.up(f_64_flat) # (BT, d_16, H/16, W/16) + del f_64_flat + f_16_up = f_16_up.reshape(B, T, self.d_16, H // 16, W // 16) + + # Learned-weight U-Net skip: add encoder /16 features scaled by a + # per-channel parameter. Broadcasts (d_16,) over (B, T, C, H, W). + scale = self.uskip_scale.view(1, 1, self.d_16, 1, 1) + f_16_dec = f_16_up + scale * f_16_mid + del f_16_up, f_16_mid + + # /16 decoder side (Swin-3D) + baton = self._swin3d_run( + self.dec_16, f_16_dec, t_stride, + late_skip_5d=f_16_patch, + late_skip_scale=self.patch_late_skip_scale, + late_skip_index=max(len(self.dec_16) - 2, 0), + ) # (B, T, d_16, H/16, W/16) + del f_16_dec, f_16_patch + + # Aux head: /16 baton → /2 pred (α, FG). With production dims and + # non-overlap patching, FG is the exact area-downsampled inverse of + # the stem RGB patch basis at init; alpha remains zero. + BT = B * T + baton_flat = baton.reshape(BT, self.d_16, H // 16, W // 16) + aux_raw = self.aux_depatchify(baton_flat) # (BT, 4, H/2, W/2) + del baton_flat + aux_out = aux_raw.reshape(B, T, 4, H // 2, W // 2) + del aux_raw + + out = {'baton': baton, 'aux': aux_out, 'bottleneck': bottleneck_out} + if recurrent_states_out is not None: + out['recurrent_states'] = tuple(recurrent_states_out) + return out diff --git a/corridorkey/stage2.py b/corridorkey/stage2.py new file mode 100644 index 0000000000000000000000000000000000000000..2b2bf1e6853d0aaeb3fea0d9e4dd809025f2874f --- /dev/null +++ b/corridorkey/stage2.py @@ -0,0 +1,532 @@ +"""Stage 2 — pure-transformer refinement at /16, depatchify to /1. + +Design: + • Patchify RGB to 768d at /16. + • Concatenate stage-1 baton (d=512) with the 768d RGB patch stream: + features = [baton | rgb_patchify]. + • 8 × mixed FullSpatial/Swin/CSwinBlock2D spatial attention (d=1280): + full spatial at the first and middle refresh points, plus Swin/CSwin + local/axial refinement across the four half-window shift phases. + • In the main training phase, a SECOND zero-inited 768d patchify_2 value + branch of raw RGB is added one block before the end. A live full-rank + gate reads the mixed token stream and modulates that zero-init value, + so the branch is output-neutral at insertion while still becoming + context-dependent once it starts learning. MAE pretraining omits this + module; it is introduced zero-init when switching to the main phase. + • 2 more SwinBlock2D blocks. + • Depatchify 16× (Linear(d → 4·16²) + PixelShuffle(16)); FG rows are + initialized as the inverse of the RGB patch basis when the exact 768d + RGB basis is available, and alpha rows start at zero. + +No temporal processing here — stage 1 has done all temporal work. Stage 2 +runs per-frame (B·T batched), which also means windowed spatial attention +is the tightest-possible attention shape for a 16K-token-per-frame budget +(256 windows × 64 tokens squared = 1M entries per head per layer). +""" + +import torch +import torch.nn as nn +import torch.nn.functional as F + +from .blocks import ( + PatchifyConv, Depatchify, FullSpatialBlock2D, SwinBlock2D, CSwinBlock2D, + LayerNorm2d, icnr_init_, init_orthogonal_rgb_passthrough_, +) +from .rope import RoPE2D + + +def _frontloaded_qkv_kernels(total: int, kernel: int, + fraction: float = 1.0) -> list[int]: + kernel = int(kernel or 0) + total = int(total) + if total <= 0 or kernel <= 0: + return [0 for _ in range(max(total, 0))] + frac = max(0.0, min(float(fraction), 1.0)) + active = total if frac >= 1.0 else int(round(total * frac)) + active = max(0, min(total, active)) + return [kernel if i < active else 0 for i in range(total)] + + +def _stage2_qkv_kernels(total: int, kernel: int, + fraction: float = 1.0) -> list[int]: + kernel = int(kernel or 0) + total = int(total) + if total <= 0 or kernel <= 0: + return [0 for _ in range(max(total, 0))] + frac = max(0.0, min(float(fraction), 1.0)) + if frac <= 0.0: + return [0 for _ in range(total)] + + if total != 10: + return _frontloaded_qkv_kernels(total, kernel, frac) + + # Default stage-2 layout with full-spatial refresh blocks: + # 0 Full, 1 Swin(h,h), 2 CSwin, 3 Swin(h,0), + # 4 Full, 5 CSwin, 6 Swin(0,0), 7 Swin(h,h), + # 8 Swin(h,0), 9 Swin(0,h) + # Prefer QKV-DW on Swin blocks, one pass per shift phase first. + if abs(frac - 0.25) < 1e-12: + active = 4 + else: + active = total if frac >= 1.0 else int(round(total * frac)) + active = max(0, min(total, active)) + + priority = [1, 3, 6, 9, 7, 8, 0, 4, 2, 5] + plan = [0 for _ in range(total)] + for i in priority[:active]: + plan[i] = kernel + return plan + + +def _qkv_kernel_at(kernels, i: int) -> int: + if isinstance(kernels, (list, tuple)): + return int(kernels[i]) if i < len(kernels) else 0 + return int(kernels or 0) + + +def _stage2_pre_blocks(d, num_heads, window_size, mlp_mult, rope, + num_pre_blocks=8, + local_mix_kernel=0, + local_mix_init_std=None, + mlp_dw_kernel=0, + qkv_dw_kernel=0): + """Build the pre-patch2 pattern. + + An earlier layout used 6 blocks: 2 dense → 2 CSwin → 2 dense with orthogonal shifts. + The current layout uses 8 blocks: 2 dense → CSwin → 2 inserted dense → CSwin → 2 dense. + + The CSwin pair in the middle gives two full-global sweeps (each block + mixes both H and W axes via parallel head groups), with a local Swin + refinement pair between those sweeps. Dense Swin shifts are arranged so + the 8 pre/final dense blocks cover the four half-window phases evenly. + """ + w = window_size + hw = w // 2 + if num_pre_blocks == 6: + specs = [ + ("swin", (0, 0)), + ("swin", (hw, hw)), + ("cswin", None), + ("cswin", None), + ("swin", (hw, 0)), + ("swin", (0, hw)), + ] + elif num_pre_blocks == 8: + specs = [ + ("full", None), + ("swin", (hw, hw)), + ("cswin", None), + ("swin", (hw, 0)), + ("full", None), + ("cswin", None), + ("swin", (0, 0)), + ("swin", (hw, hw)), + ] + else: + raise ValueError( + f"num_pre_blocks must be 6 or 8 for the fixed CSwin pattern; got {num_pre_blocks}") + blocks = [] + for i, (kind, shift) in enumerate(specs): + k = _qkv_kernel_at(qkv_dw_kernel, i) + if kind == "full": + blocks.append( + FullSpatialBlock2D(d, num_heads, + mlp_mult=mlp_mult, rope=rope, + local_mix_kernel=local_mix_kernel, + local_mix_init_std=local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=k) + ) + elif kind == "swin": + blocks.append( + SwinBlock2D(d, num_heads, w, shift=shift, + mlp_mult=mlp_mult, rope=rope, + local_mix_kernel=local_mix_kernel, + local_mix_init_std=local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=k) + ) + else: + blocks.append( + CSwinBlock2D(d, num_heads, mlp_mult=mlp_mult, rope=rope, + local_mix_kernel=local_mix_kernel, + local_mix_init_std=local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=k) + ) + return blocks + + +def _stage2_final_blocks(d, num_heads, window_size, mlp_mult, rope, + local_mix_kernel=0, + local_mix_init_std=None, + mlp_dw_kernel=0, + qkv_dw_kernel=0): + """Build the 2-block post-patch2 pattern: dense (w/2,0) → dense (0,w/2). + Runs AFTER the late zero-init RGB pixel-skip (patchify_2) is added, so + fresh pixel detail immediately crosses both seam axes.""" + w = window_size + hw = w // 2 + return [ + SwinBlock2D(d, num_heads, w, shift=(hw, 0), + mlp_mult=mlp_mult, rope=rope, + local_mix_kernel=local_mix_kernel, + local_mix_init_std=local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=_qkv_kernel_at(qkv_dw_kernel, 0)), + SwinBlock2D(d, num_heads, w, shift=(0, hw), + mlp_mult=mlp_mult, rope=rope, + local_mix_kernel=local_mix_kernel, + local_mix_init_std=local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=_qkv_kernel_at(qkv_dw_kernel, 1)), + ] + + +class Refine4ConvNeXtBlock(nn.Module): + """Small ConvNeXt-style block over /4 feature maps. + + NCHW is kept throughout so cuDNN sees ordinary conv tensors. The branch is + residual, but not zero-initialized; the enclosing residual-output projection + is zero-init, so enabling the head is still output-neutral at insertion. + """ + + def __init__(self, width: int, kernel_size: int = 7, + expansion: float = 2.0): + super().__init__() + if kernel_size <= 0 or kernel_size % 2 != 1: + raise ValueError( + f"kernel_size must be a positive odd integer; got {kernel_size}") + hidden = int(round(width * float(expansion))) + self.pad = kernel_size // 2 + self.dw = nn.Conv2d(width, width, kernel_size=kernel_size, + padding=0, groups=width, bias=True) + self.norm = LayerNorm2d(width) + self.pw1 = nn.Conv2d(width, hidden, kernel_size=1, bias=True) + self.pw2 = nn.Conv2d(hidden, width, kernel_size=1, bias=True) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + y = x + if self.pad: + mode = 'reflect' if x.shape[-2] > self.pad and x.shape[-1] > self.pad else 'replicate' + y = F.pad(y, (self.pad, self.pad, self.pad, self.pad), mode=mode) + y = self.dw(y) + y = self.norm(y) + y = self.pw2(F.gelu(self.pw1(y), approximate='tanh')) + return x + y + + +class Stage2Refine4Head(nn.Module): + """Optional /4 conv refinement head for the final stage-2 output. + + Inputs: + * mixed /16 stage-2 tokens after the final transformer block; + * exact RGB pixel-unshuffle to /4 (3 * 4 * 4 = 48 channels). + + Output is a full-res residual over the normal depatchified stage-2 logits. + The residual projection is zero-initialized, so the head is an exact no-op + until it learns. + """ + + def __init__( + self, + token_dim: int, + token_channels: int = 48, + width: int = 96, + blocks: int = 2, + kernel_size: int = 7, + expansion: float = 2.0, + gradient_checkpoint: bool = False, + ): + super().__init__() + if token_channels <= 0: + raise ValueError("token_channels must be positive") + if width <= 0: + raise ValueError("width must be positive") + if blocks <= 0: + raise ValueError("blocks must be positive") + self.token_channels = int(token_channels) + self.rgb_channels = 3 * 4 * 4 + self.width = int(width) + self.gradient_checkpoint = bool(gradient_checkpoint) + + self.token_up = nn.Conv2d( + token_dim, self.token_channels * 4 * 4, + kernel_size=1, bias=True, + ) + icnr_init_(self.token_up, upscale_factor=4) + + in_channels = self.token_channels + self.rgb_channels + self.in_proj = ( + nn.Identity() if in_channels == self.width + else nn.Conv2d(in_channels, self.width, kernel_size=1, bias=True) + ) + self.blocks = nn.ModuleList([ + Refine4ConvNeXtBlock( + self.width, kernel_size=kernel_size, expansion=expansion) + for _ in range(int(blocks)) + ]) + self.out = nn.Conv2d(self.width, 4 * 4 * 4, kernel_size=1, bias=True) + nn.init.zeros_(self.out.weight) + nn.init.zeros_(self.out.bias) + + def forward(self, rgb_flat: torch.Tensor, + feat_ch_first: torch.Tensor) -> torch.Tensor: + token = F.pixel_shuffle(self.token_up(feat_ch_first), 4) + rgb = F.pixel_unshuffle(rgb_flat, 4) + x = torch.cat([token, rgb], dim=1) + x = self.in_proj(x) + for block in self.blocks: + if self.gradient_checkpoint and self.training and torch.is_grad_enabled(): + x = torch.utils.checkpoint.checkpoint( + block, x, use_reentrant=False) + else: + x = block(x) + return F.pixel_shuffle(self.out(x), 4) + + +class Stage2(nn.Module): + def __init__( + self, + d: int = 1280, + d_baton: int = 512, + num_heads: int = 16, # 1280 / 16 = 80 (head_dim) + num_blocks: int = 10, + num_final_blocks: int = 2, + window_size: int = 16, + mlp_mult: float = 2.5, + # Per-block gradient checkpointing during training — each Swin-2D + # block individually wrapped so only one block's activations are + # resident at a time during backward. + gradient_checkpoint: bool = False, + # Overlap patchify/depatchify (see blocks.PatchifyConv / Depatchify). + # overlap_patchify is a legacy bool alias (True → mult=3). + # overlap_mult: 1 (no overlap) | 2 (half) | 3 (full, 3×3 neighborhood). + overlap_patchify: bool = False, + overlap_mult: int = 2, + depatch_fourier_features: int = 0, + depatch_fourier_kernel: int = 3, + # MAE-style pretraining hooks. Disabled in the normal inference path. + enable_mae_mask_tokens: bool = False, + # Optional depthwise local mixer inserted after attention and before + # MLP in every stage-2 Swin/CSwin block. kernel=0 disables. + local_mix_kernel: int = 0, + local_mix_init_std: float | None = None, + mlp_dw_kernel: int = 0, + qkv_dw_kernel: int = 0, + qkv_dw_fraction: float = 1.0, + refine4_blocks: int = 0, + refine4_width: int = 96, + refine4_token_channels: int = 48, + refine4_kernel: int = 7, + refine4_expansion: float = 2.0, + # RoPE augmentations (DINOv3-style) — see rope.py. + rope_shift_coords: float = None, + rope_jitter_coords: float = None, + rope_rescale_coords: float = None, + rope_base: float = 100.0, + ): + super().__init__() + assert num_blocks >= 2, ( + f"num_blocks={num_blocks} too few — need ≥2 so the zero-init " + f"pixel-skip can fire at least one block before the end.") + assert 1 <= num_final_blocks < num_blocks, ( + f"num_final_blocks={num_final_blocks} must be in [1, num_blocks-1]; " + f"num_blocks={num_blocks}") + assert 0 < d_baton < d, ( + f"d_baton={d_baton} must be in (0, d={d}) exclusive") + self.d = d + self.d_baton = d_baton + self.d_rgb = d - d_baton + self.num_blocks = num_blocks + self.num_final_blocks = num_final_blocks + self.num_pre_blocks = num_blocks - num_final_blocks + self.window_size = window_size + self.gradient_checkpoint = gradient_checkpoint + self.enable_mae_mask_tokens = bool(enable_mae_mask_tokens) + self.use_patch_2 = not self.enable_mae_mask_tokens + self.refine4_enabled = int(refine4_blocks) > 0 + qkv_plan = _stage2_qkv_kernels( + num_blocks, + qkv_dw_kernel, + qkv_dw_fraction, + ) + qkv_pre = qkv_plan[:self.num_pre_blocks] + qkv_final = qkv_plan[self.num_pre_blocks:] + self.qkv_dw_active_blocks = sum(1 for k in qkv_plan if k > 0) + self.qkv_dw_total_blocks = len(qkv_plan) + + # Patchify_1: 3 → d_rgb at /16, concatenated after the baton. + self.patch_1 = PatchifyConv(3, self.d_rgb, patch=16, + zero_init_first=0, + overlap=overlap_patchify, + overlap_mult=overlap_mult) + + # Patchify_2 (late pixel skip): main-training only. Pretraining + # omits it so full-RGB reconstruction cannot lean on a second raw-RGB + # tokenization path. In phase 2, the value branch starts at zero and + # a live gate reads the already-mixed token stream directly so token + # magnitude remains available as a precision/confidence signal. + if self.use_patch_2: + self.patch_2 = PatchifyConv(3, self.d_rgb, patch=16, + zero_init_all=True, + overlap=overlap_patchify, + overlap_mult=overlap_mult) + self.patch_2_gate = nn.Linear(d, self.d_rgb, bias=True) + nn.init.xavier_uniform_(self.patch_2_gate.weight, gain=0.25) + nn.init.zeros_(self.patch_2_gate.bias) + + # Shared RoPE across Swin blocks. + head_dim = d // num_heads + self.rope = RoPE2D(head_dim, + base=rope_base, + shift_coords=rope_shift_coords, + jitter_coords=rope_jitter_coords, + rescale_coords=rope_rescale_coords) + + # Block layout: + # pre_blocks (8): full-spatial, dense (w/2,w/2), CSwin, + # inserted dense (w/2,0), full-spatial, + # CSwin, dense (0,0), dense (w/2,w/2) + # → patchify_2 zero-init RGB skip added here + # final_blocks (2): dense (w/2,0), dense (0,w/2) + assert self.num_pre_blocks in (6, 8), ( + f"num_pre_blocks must be 6 or 8 for the CSwin-in-the-middle pattern; " + f"got {self.num_pre_blocks} (num_blocks={num_blocks}, " + f"num_final_blocks={num_final_blocks})") + assert num_final_blocks == 2, ( + f"num_final_blocks must be 2 for the 2-phase final stack; " + f"got {num_final_blocks}") + self.pre_blocks = nn.ModuleList( + _stage2_pre_blocks(d, num_heads, window_size, mlp_mult, self.rope, + num_pre_blocks=self.num_pre_blocks, + local_mix_kernel=local_mix_kernel, + local_mix_init_std=local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=qkv_pre) + ) + self.final_blocks = nn.ModuleList( + _stage2_final_blocks(d, num_heads, window_size, mlp_mult, self.rope, + local_mix_kernel=local_mix_kernel, + local_mix_init_std=local_mix_init_std, + mlp_dw_kernel=mlp_dw_kernel, + qkv_dw_kernel=qkv_final) + ) + + # Depatchify: d → 4 alpha/RGB channels at /1 via PixelShuffle(16). + # When dims are the production 768d RGB patch basis, FG rows invert + # patch_1 exactly and alpha/baton rows do not affect output at init. + self.depatchify = Depatchify(d, out_channels=4, r=16, + zero_init_all=True, bias=True, + overlap=overlap_patchify, + overlap_mult=overlap_mult, + fourier_features=depatch_fourier_features, + fourier_kernel=depatch_fourier_kernel) + self.rgb_passthrough_init = init_orthogonal_rgb_passthrough_( + self.patch_1, self.depatchify, + token_offset=self.d_baton, + output_rgb_offset=1, + downsample=1, + ) + self.refine4 = ( + Stage2Refine4Head( + d, token_channels=refine4_token_channels, + width=refine4_width, blocks=int(refine4_blocks), + kernel_size=refine4_kernel, expansion=refine4_expansion, + gradient_checkpoint=gradient_checkpoint, + ) + if self.refine4_enabled else None + ) + if self.enable_mae_mask_tokens: + self.mae_rgb_token = nn.Parameter(torch.zeros(self.d_rgb)) + + # ----------------------------------------------------------------- + + def _apply_mae_rgb_token( + self, + x: torch.Tensor, + mae_mask_16: torch.Tensor | None, + B: int, + T: int, + name: str, + ) -> torch.Tensor: + if mae_mask_16 is None: + return x + if not self.enable_mae_mask_tokens: + raise RuntimeError(f"{name} received an MAE mask but MAE tokens are disabled") + BT, C, H16, W16 = x.shape + if C != self.d_rgb: + raise ValueError(f"{name} channel count {C} != d_rgb {self.d_rgb}") + expected = (B, T, 1, H16, W16) + if tuple(mae_mask_16.shape) != expected: + raise ValueError( + f"{name} mask shape {tuple(mae_mask_16.shape)} != expected {expected}") + mask = mae_mask_16.reshape(BT, 1, H16, W16).to(device=x.device, + dtype=torch.bool) + token = self.mae_rgb_token.to(device=x.device, dtype=x.dtype).view(1, C, 1, 1) + return torch.where(mask, token, x) + + def forward( + self, + rgb: torch.Tensor, # (B, T, 3, H, W) signed asinh encoded scene-linear RGB + baton: torch.Tensor, # (B, T, d_baton, H/16, W/16) from stage 1 + mae_mask_16: 'torch.Tensor | None' = None, # (B,T,1,H/16,W/16) + ) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]: + """Returns logits (B, T, 4, H, W), or (refined, base) when refine4 is on.""" + B, T, _, H, W = rgb.shape + assert H % 16 == 0 and W % 16 == 0, ( + f"Stage2: H,W must be /16-aligned; got {(H, W)}") + assert baton.shape == (B, T, self.d_baton, H // 16, W // 16), ( + f"baton shape mismatch: got {baton.shape}, " + f"expected {(B, T, self.d_baton, H // 16, W // 16)}") + + BT = B * T + rgb_flat = rgb.reshape(BT, 3, H, W) + + # Patchify_1 (RGB → d_rgb at /16), then concatenate after the baton. + rgb_feat = self.patch_1(rgb_flat) # (BT, d_rgb, H/16, W/16) + rgb_feat = self._apply_mae_rgb_token( + rgb_feat, mae_mask_16, B, T, "stage2 patch_1") + + baton_flat = baton.reshape(BT, self.d_baton, H // 16, W // 16) + feat = torch.cat([baton_flat, rgb_feat], dim=1) # (BT, d, H/16, W/16) + + # Swin-2D permute to (BT, H/16, W/16, d) + x = feat.permute(0, 2, 3, 1).contiguous() + + # Per-block gradient checkpointing: only one block's activations + # are resident at a time during backward when gradient_checkpoint is on. + def run_block(blk, x_in): + if self.gradient_checkpoint and self.training and torch.is_grad_enabled(): + return torch.utils.checkpoint.checkpoint( + blk, x_in, use_reentrant=False, + ) + return blk(x_in) + + # Pre-patch_2 blocks (num_pre_blocks) + for blk in self.pre_blocks: + x = run_block(blk, x) + + if self.use_patch_2: + # Late pixel skip: zero-init patchify_2 of raw RGB, modulated by + # a full-rank gate over the mixed token stream. The value branch + # is exactly zero at insertion, so the live zero-bias gate only + # affects gradients until patch_2 wakes up. + gate = F.silu(self.patch_2_gate(x)) # (BT, H/16, W/16, d_rgb) + patch2 = self.patch_2(rgb_flat) # (BT, d_rgb, H/16, W/16) + patch2 = patch2.permute(0, 2, 3, 1).contiguous() * gate + x = x + F.pad(patch2, (self.d_baton, 0)) + + # Post-patch_2 final blocks (num_final_blocks) + for blk in self.final_blocks: + x = run_block(blk, x) + + # Depatchify to /1. With production dims and non-overlap patching, + # FG rows are the exact inverse of patch_1 at init; alpha rows and + # baton columns are zero. + feat_ch_first = x.permute(0, 3, 1, 2).contiguous() # (BT, d, H/16, W/16) + out_raw = self.depatchify(feat_ch_first) # (BT, 4, H, W) + if self.refine4 is not None: + refined = out_raw + self.refine4(rgb_flat, feat_ch_first) + return refined.reshape(B, T, 4, H, W), out_raw.reshape(B, T, 4, H, W) + return out_raw.reshape(B, T, 4, H, W) diff --git a/corridorkey/video_io.py b/corridorkey/video_io.py new file mode 100644 index 0000000000000000000000000000000000000000..16f952b5351104d5608d8e52b6cc78c247aff607 --- /dev/null +++ b/corridorkey/video_io.py @@ -0,0 +1,820 @@ +from __future__ import annotations + +import glob +import math +import os +import queue +import shutil +import subprocess +import threading +from dataclasses import dataclass +from pathlib import Path + +os.environ.setdefault("OPENCV_IO_ENABLE_OPENEXR", "1") + +import cv2 +import numpy as np +import torch + +from . import color_utils as cu + + +@dataclass +class WriterConfig: + ffmpeg_bin: str = "ffmpeg" + codec: str = "libx264" + crf: int = 12 + preset: str = "medium" + pix_fmt: str = "yuv444p10le" + bitrate: str | None = None + threads: int = 2 + + +def require_ffmpeg(ffmpeg_bin: str) -> None: + if shutil.which(ffmpeg_bin) is None: + raise FileNotFoundError( + f"ffmpeg binary not found: {ffmpeg_bin}. " + "Install ffmpeg or pass --ffmpeg_bin.") + + +def safe_remove(path: str | os.PathLike | None) -> None: + if not path: + return + try: + os.remove(path) + except FileNotFoundError: + pass + + +def none_path(path: str | None) -> str | None: + if path is None: + return None + text = str(path).strip() + if text.lower() in ("", "none", "null", "off", "false", "-"): + return None + return str(path) + + +def raw_clip_name(path: str | os.PathLike) -> str: + norm = os.path.normpath(str(path)) + name = os.path.splitext(os.path.basename(norm))[0] + if name.lower() in {"input", "frames", "rgb"}: + return os.path.basename(os.path.dirname(norm)) or name + return name + + +def frame_paths(frame_dir: str | os.PathLike) -> list[str]: + paths: list[str] = [] + for ext in ("*.exr", "*.png", "*.jpg", "*.jpeg"): + paths.extend(glob.glob(os.path.join(str(frame_dir), ext))) + return sorted(paths) + + +def ceil_to_multiple(x: int, q: int) -> int: + return max(q, ((int(x) + q - 1) // q) * q) + + +def scaled_hw(h: int, w: int, scale: float) -> tuple[int, int]: + scale = float(scale) + if scale <= 0: + raise ValueError(f"scale must be > 0, got {scale}") + return max(1, int(round(h * scale))), max(1, int(round(w * scale))) + + +def resize_hwc_lanczos(img: np.ndarray, hw: tuple[int, int]) -> np.ndarray: + h, w = hw + if img.shape[:2] == (h, w): + return img + return cv2.resize(img, (w, h), interpolation=cv2.INTER_LANCZOS4) + + +def resize_chw_lanczos(x: np.ndarray, hw: tuple[int, int]) -> np.ndarray: + h, w = hw + if x.shape[-2:] == (h, w): + return x + if x.shape[0] == 1: + y = cv2.resize(x[0], (w, h), interpolation=cv2.INTER_LANCZOS4) + return y[None].astype(x.dtype, copy=False) + y = cv2.resize(np.moveaxis(x, 0, -1), (w, h), + interpolation=cv2.INTER_LANCZOS4) + return np.moveaxis(y, -1, 0).astype(x.dtype, copy=False) + + +def center_pad_to_shape(img: np.ndarray, h_out: int, w_out: int): + h, w = img.shape[:2] + pad_t = (h_out - h) // 2 + pad_b = h_out - h - pad_t + pad_l = (w_out - w) // 2 + pad_r = w_out - w - pad_l + if pad_t or pad_b or pad_l or pad_r: + img = cv2.copyMakeBorder( + img, pad_t, pad_b, pad_l, pad_r, cv2.BORDER_REFLECT_101) + return img, (pad_l, pad_r, pad_t, pad_b) + + +def crop_chw_to_native(x: np.ndarray, native_hw: tuple[int, int], pad): + h, w = native_hw + pad_l, _, pad_t, _ = pad + return x[..., pad_t:pad_t + h, pad_l:pad_l + w] + + +def crop_model_chw_to_output(x: np.ndarray, model_native_hw, pad, output_hw): + cropped = crop_chw_to_native(x, model_native_hw, pad) + return resize_chw_lanczos(cropped, output_hw) + + +def read_image_rgb(path: str | os.PathLike): + path = str(path) + if path.lower().endswith(".exr"): + img = cv2.imread(path, cv2.IMREAD_UNCHANGED) + if img is None: + raise FileNotFoundError(path) + img = img.astype(np.float32, copy=False) + if img.ndim == 3 and img.shape[2] == 4: + img = img[:, :, :3] + if img.ndim == 3: + img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) + return img + img = cv2.imread(path, cv2.IMREAD_COLOR) + if img is None: + raise FileNotFoundError(path) + return cv2.cvtColor(img, cv2.COLOR_BGR2RGB).astype(np.float32) / 255.0 + + +def read_hint_image(path: str | os.PathLike) -> np.ndarray: + img = cv2.imread(str(path), cv2.IMREAD_UNCHANGED) + if img is None: + raise FileNotFoundError(path) + if img.dtype == np.uint8: + img = img.astype(np.float32) / 255.0 + elif img.dtype == np.uint16: + img = img.astype(np.float32) / 65535.0 + else: + img = img.astype(np.float32) + if img.ndim == 3: + img = img[:, :, 3] if img.shape[2] == 4 else img[:, :, 0] + return np.clip(img, 0.0, 1.0) + + +def has_linear_adjust(meta: dict) -> bool: + return ( + abs(float(meta.get("linear_brightness", 1.0)) - 1.0) > 1e-6 + or abs(float(meta.get("linear_contrast", 1.0)) - 1.0) > 1e-6 + ) + + +def apply_linear_adjust(linear: np.ndarray, meta: dict) -> np.ndarray: + gain = float(meta.get("linear_brightness", 1.0)) + contrast = float(meta.get("linear_contrast", 1.0)) + if abs(gain - 1.0) <= 1e-6 and abs(contrast - 1.0) <= 1e-6: + return linear + pivot = float(meta.get("linear_contrast_pivot", 0.18)) + return gain * (pivot + contrast * (linear - pivot)) + + +def invert_linear_adjust(linear: np.ndarray, meta: dict) -> np.ndarray: + gain = float(meta.get("linear_brightness", 1.0)) + contrast = float(meta.get("linear_contrast", 1.0)) + if abs(gain - 1.0) <= 1e-6 and abs(contrast - 1.0) <= 1e-6: + return linear + pivot = float(meta.get("linear_contrast_pivot", 0.18)) + return (linear / gain - pivot) / contrast + pivot + + +def probe_video_meta(video_path: str, *, num_frames: int, shape_quantum: int, + native_resolution: bool, start_mode: str, seed: int, + model_scale: float, linear_brightness: float, + linear_contrast: float, + linear_contrast_pivot: float) -> dict: + cap = cv2.VideoCapture(video_path) + if not cap.isOpened(): + raise RuntimeError(f"Could not open video: {video_path}") + native_fps = cap.get(cv2.CAP_PROP_FPS) or 24.0 + w_native = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) + h_native = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) + n_total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) + cap.release() + + q = int(shape_quantum) + if native_resolution: + h_model, w_model = scaled_hw(h_native, w_native, model_scale) + h_out = ceil_to_multiple(h_model, q) + w_out = ceil_to_multiple(w_model, q) + pad_l = (w_out - w_model) // 2 + pad_r = w_out - w_model - pad_l + pad_t = (h_out - h_model) // 2 + pad_b = h_out - h_model - pad_t + output_hw = (h_native, w_native) + native_hw = (h_native, w_native) + else: + raise ValueError("non-native inference is intentionally not exposed") + + n_to_load = n_total if num_frames <= 0 else min(num_frames, n_total) + start_frame = 0 + max_start = max(n_total - n_to_load, 0) + if max_start > 0 and num_frames > 0: + if start_mode == "middle": + start_frame = max_start // 2 + elif start_mode == "random_middle": + lo = max_start // 4 + hi = (3 * max_start) // 4 + start_frame = int(np.random.default_rng(int(seed)).integers(lo, hi + 1)) + elif start_mode != "begin": + raise ValueError(f"unknown start mode {start_mode!r}") + + meta = { + "native_fps": native_fps, + "native_hw": native_hw, + "output_hw": output_hw, + "model_native_hw": (h_model, w_model), + "model_scale": float(model_scale), + "source_hw": (h_native, w_native), + "shape_hw": (h_out, w_out), + "start_frame": start_frame, + "num_frames": n_to_load, + "pad": (pad_l, pad_r, pad_t, pad_b), + "native_resolution": True, + "linear_brightness": float(linear_brightness), + "linear_contrast": float(linear_contrast), + "linear_contrast_pivot": float(linear_contrast_pivot), + } + print(f" probed {n_to_load} frames @ {w_out}x{h_out} " + f"(native {w_native}x{h_native}, fps {native_fps:.2f}, " + f"start={start_frame}, pad={meta['pad']})") + return meta + + +def _read_frame_dir_views(path: str, *, exr_linear: bool): + raw = read_image_rgb(path) + if exr_linear: + linear = raw.astype(np.float32, copy=False) + display = np.clip(cu.linear_to_srgb(np.maximum(linear, 0.0)), 0.0, 1.0) + else: + display = np.clip(raw.astype(np.float32, copy=False), 0.0, 1.0) + linear = cu.srgb_to_linear(display) + return display, linear + + +def probe_frame_dir_meta(frame_dir: str, *, num_frames: int, + shape_quantum: int, fps: float, + exr_linear: bool, model_scale: float) -> dict: + paths = frame_paths(frame_dir) + if not paths: + raise RuntimeError(f"No frames found under {frame_dir}") + if num_frames > 0: + paths = paths[:num_frames] + first_display, _ = _read_frame_dir_views(paths[0], exr_linear=exr_linear) + h_native, w_native = first_display.shape[:2] + h_model, w_model = scaled_hw(h_native, w_native, model_scale) + h_out = ceil_to_multiple(h_model, shape_quantum) + w_out = ceil_to_multiple(w_model, shape_quantum) + pad_l = (w_out - w_model) // 2 + pad_r = w_out - w_model - pad_l + pad_t = (h_out - h_model) // 2 + pad_b = h_out - h_model - pad_t + meta = { + "frame_paths": paths, + "native_fps": float(fps), + "native_hw": (h_native, w_native), + "output_hw": (h_native, w_native), + "model_native_hw": (h_model, w_model), + "model_scale": float(model_scale), + "source_hw": (h_native, w_native), + "shape_hw": (h_out, w_out), + "start_frame": 0, + "num_frames": len(paths), + "pad": (pad_l, pad_r, pad_t, pad_b), + "native_resolution": True, + "exr_linear": bool(exr_linear), + "linear_brightness": 1.0, + "linear_contrast": 1.0, + "linear_contrast_pivot": 0.18, + } + print(f" probed {len(paths)} frames from {frame_dir} @ {w_out}x{h_out} " + f"(native {w_native}x{h_native}, fps {fps:.2f}, " + f"exr_linear={exr_linear}, pad={meta['pad']})") + return meta + + +def preprocess_video_frame_views(frame_bgr, meta: dict, *, need_model: bool): + h_out, w_out = meta["shape_hw"] + h_model, w_model = meta["model_native_hw"] + frame = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB).astype(np.float32) / 255.0 + frame = resize_hwc_lanczos(frame, (h_model, w_model)) + frame, _ = center_pad_to_shape(frame, h_out, w_out) + review = torch.from_numpy(frame.transpose(2, 0, 1)).to(dtype=torch.bfloat16) + if not need_model and not has_linear_adjust(meta): + return None, review, review + linear = cu.srgb_to_linear(frame) + linear_model = apply_linear_adjust(linear, meta) + if has_linear_adjust(meta): + foundation_np = cu.linear_to_vfm_sdr(linear_model) + foundation = torch.from_numpy( + foundation_np.transpose(2, 0, 1)).to(dtype=torch.bfloat16) + else: + foundation = review + if not need_model: + return None, foundation, review + model = cu.linear_to_asinh(linear_model) + model_t = torch.from_numpy(model.transpose(2, 0, 1)).to(dtype=torch.bfloat16) + return model_t, foundation, review + + +def preprocess_frame_dir_path(path: str, meta: dict, *, need_model: bool): + h_out, w_out = meta["shape_hw"] + h_model, w_model = meta["model_native_hw"] + display, linear = _read_frame_dir_views( + path, exr_linear=bool(meta.get("exr_linear", False))) + display = resize_hwc_lanczos(display, (h_model, w_model)) + linear = resize_hwc_lanczos(linear, (h_model, w_model)) + display, _ = center_pad_to_shape(np.clip(display, 0.0, 1.0), h_out, w_out) + display_t = torch.from_numpy( + display.transpose(2, 0, 1)).to(dtype=torch.bfloat16) + if not need_model: + return None, display_t, display_t + linear, _ = center_pad_to_shape(linear, h_out, w_out) + model = cu.linear_to_asinh(linear) + model_t = torch.from_numpy(model.transpose(2, 0, 1)).to(dtype=torch.bfloat16) + return model_t, display_t, display_t + + +class SequentialVideoWindowReader: + def __init__(self, video_path: str, meta: dict): + self.video_path = video_path + self.meta = meta + self.cap = cv2.VideoCapture(video_path) + if not self.cap.isOpened(): + raise RuntimeError(f"Could not open video: {video_path}") + self.start_frame = int(meta["start_frame"]) + self.n_frames = int(meta["num_frames"]) + if self.start_frame: + self.cap.set(cv2.CAP_PROP_POS_FRAMES, self.start_frame) + self.next_t = 0 + self.cache: dict[int, tuple[torch.Tensor, torch.Tensor, torch.Tensor]] = {} + + def close(self): + self.cap.release() + + def _read_until(self, t_end: int): + while self.next_t < t_end: + ret, frame = self.cap.read() + if not ret: + raise RuntimeError( + f"{self.video_path}: failed decoding frame " + f"{self.start_frame + self.next_t}") + self.cache[self.next_t] = preprocess_video_frame_views( + frame, self.meta, need_model=True) + self.next_t += 1 + + def window(self, t0: int, t1: int): + self._read_until(t1) + model = torch.stack([self.cache[t][0] for t in range(t0, t1)]) + foundation = torch.stack([self.cache[t][1] for t in range(t0, t1)]) + review = torch.stack([self.cache[t][2] for t in range(t0, t1)]) + for old_t in [k for k in self.cache if k < t0]: + del self.cache[old_t] + return model, foundation, review + + +class SequentialFrameDirWindowReader: + def __init__(self, frame_dir: str, meta: dict): + self.frame_dir = frame_dir + self.meta = meta + self.paths = meta["frame_paths"] + self.next_t = 0 + self.cache: dict[int, tuple[torch.Tensor, torch.Tensor, torch.Tensor]] = {} + + def close(self): + pass + + def _read_until(self, t_end: int): + while self.next_t < t_end: + self.cache[self.next_t] = preprocess_frame_dir_path( + self.paths[self.next_t], self.meta, need_model=True) + self.next_t += 1 + + def window(self, t0: int, t1: int): + self._read_until(t1) + model = torch.stack([self.cache[t][0] for t in range(t0, t1)]) + foundation = torch.stack([self.cache[t][1] for t in range(t0, t1)]) + review = torch.stack([self.cache[t][2] for t in range(t0, t1)]) + for old_t in [k for k in self.cache if k < t0]: + del self.cache[old_t] + return model, foundation, review + + +def iter_video_frame_chunks(video_path: str, meta: dict, *, chunk_size: int, + need_model: bool): + cap = cv2.VideoCapture(video_path) + if not cap.isOpened(): + raise RuntimeError(f"Could not open video: {video_path}") + if int(meta["start_frame"]): + cap.set(cv2.CAP_PROP_POS_FRAMES, int(meta["start_frame"])) + t = 0 + try: + while t < int(meta["num_frames"]): + model_list = [] if need_model else None + foundation_list = [] + t0 = t + while t < int(meta["num_frames"]) and len(foundation_list) < chunk_size: + ret, frame = cap.read() + if not ret: + break + model, foundation, _review = preprocess_video_frame_views( + frame, meta, need_model=need_model) + if need_model: + model_list.append(model) + foundation_list.append(foundation) + t += 1 + if not foundation_list: + break + yield t0, (torch.stack(model_list) if need_model else None), torch.stack(foundation_list) + finally: + cap.release() + if t < int(meta["num_frames"]): + raise RuntimeError( + f"{video_path}: decoded only {t}/{meta['num_frames']} requested frames") + + +def iter_frame_dir_chunks(frame_dir: str, meta: dict, *, chunk_size: int, + need_model: bool): + paths = meta["frame_paths"] + t = 0 + while t < len(paths): + t0 = t + model_list = [] if need_model else None + foundation_list = [] + while t < len(paths) and len(foundation_list) < chunk_size: + model, foundation, _review = preprocess_frame_dir_path( + paths[t], meta, need_model=need_model) + if need_model: + model_list.append(model) + foundation_list.append(foundation) + t += 1 + yield t0, (torch.stack(model_list) if need_model else None), torch.stack(foundation_list) + + +def preprocess_hint_frame(frame_bgr, meta: dict) -> torch.Tensor: + h_out, w_out = meta["shape_hw"] + h_model, w_model = meta["model_native_hw"] + if frame_bgr is None: + raise RuntimeError("hint frame decode failed") + if frame_bgr.ndim == 3: + hint = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2GRAY) + else: + hint = frame_bgr + if hint.dtype == np.uint8: + hint = hint.astype(np.float32) / 255.0 + elif hint.dtype == np.uint16: + hint = hint.astype(np.float32) / 65535.0 + else: + hint = hint.astype(np.float32) + if hint.shape[:2] != (h_model, w_model): + hint = cv2.resize(hint, (w_model, h_model), + interpolation=cv2.INTER_LANCZOS4) + hint, _ = center_pad_to_shape(np.clip(hint, 0.0, 1.0), h_out, w_out) + return torch.from_numpy(hint[None]).to(dtype=torch.bfloat16) + + +def load_initial_hint_frame(path_or_dir: str, meta: dict) -> torch.Tensor: + paths = frame_paths(path_or_dir) if os.path.isdir(path_or_dir) else [path_or_dir] + paths = [p for p in paths if os.path.isfile(p)] + if not paths: + raise RuntimeError(f"No hint frame found at {path_or_dir}") + hint = read_hint_image(paths[0]) + native_hw = meta["native_hw"] + model_hw = meta["model_native_hw"] + if hint.shape[:2] == native_hw and model_hw != native_hw: + hint = cv2.resize(hint, (model_hw[1], model_hw[0]), + interpolation=cv2.INTER_LANCZOS4) + elif hint.shape[:2] != model_hw: + raise ValueError( + f"{paths[0]}: hint shape {hint.shape[:2]} does not match " + f"native {native_hw} or model {model_hw}") + hint, pad_check = center_pad_to_shape( + hint, meta["shape_hw"][0], meta["shape_hw"][1]) + if pad_check != meta["pad"]: + raise ValueError(f"{paths[0]}: hint pad {pad_check} != input pad {meta['pad']}") + print(f" loaded initial frame-1 hint from {paths[0]}") + return torch.from_numpy(hint[None]).to(dtype=torch.bfloat16) + + +class SequentialHintVideoWindowReader: + def __init__(self, hint_video_path: str, meta: dict): + self.hint_video_path = hint_video_path + self.meta = meta + self.cap = cv2.VideoCapture(hint_video_path) + if not self.cap.isOpened(): + raise RuntimeError(f"Could not open hint video: {hint_video_path}") + self.start_frame = int(meta["start_frame"]) + self.n_frames = int(meta["num_frames"]) + hint_n_frames = int(self.cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0) + required_frames = self.start_frame + self.n_frames + if hint_n_frames > 0 and hint_n_frames < required_frames: + raise ValueError( + f"Hint video {hint_video_path} has only {hint_n_frames} frames, " + f"but the source video requires {required_frames} frames " + f"(start={self.start_frame}, num={self.n_frames})") + if self.start_frame: + self.cap.set(cv2.CAP_PROP_POS_FRAMES, self.start_frame) + self.next_t = 0 + self.cache: dict[int, torch.Tensor] = {} + + def close(self): + self.cap.release() + + def _read_until(self, t_end: int): + while self.next_t < t_end: + ret, frame = self.cap.read() + if not ret: + raise RuntimeError( + f"{self.hint_video_path}: failed decoding hint frame " + f"{self.start_frame + self.next_t}") + self.cache[self.next_t] = preprocess_hint_frame(frame, self.meta) + self.next_t += 1 + + def window(self, t0: int, t1: int): + self._read_until(t1) + hint = torch.stack([self.cache[t] for t in range(t0, t1)]) + for old_t in [k for k in self.cache if k < t0]: + del self.cache[old_t] + return hint + + +class SequentialHintFrameDirWindowReader: + def __init__(self, hint_dir_path: str, meta: dict): + self.hint_dir_path = hint_dir_path + self.meta = meta + self.paths = frame_paths(hint_dir_path) + if len(self.paths) < int(meta["num_frames"]): + raise RuntimeError( + f"{hint_dir_path}: only {len(self.paths)} hint frames for " + f"{meta['num_frames']} source frames") + self.next_t = 0 + self.cache: dict[int, torch.Tensor] = {} + + def close(self): + pass + + def _read_until(self, t_end: int): + while self.next_t < t_end: + frame = cv2.imread(self.paths[self.next_t], cv2.IMREAD_UNCHANGED) + self.cache[self.next_t] = preprocess_hint_frame(frame, self.meta) + self.next_t += 1 + + def window(self, t0: int, t1: int): + self._read_until(t1) + hint = torch.stack([self.cache[t] for t in range(t0, t1)]) + for old_t in [k for k in self.cache if k < t0]: + del self.cache[old_t] + return hint + + +def make_hint_reader(path: str | None, meta: dict): + path = none_path(path) + if path is None: + return None + if os.path.isdir(path): + return SequentialHintFrameDirWindowReader(path, meta) + return SequentialHintVideoWindowReader(path, meta) + + +def window_starts(T_all: int, chunk: int, stride: int) -> list[int]: + if T_all <= chunk: + return [0] + starts = list(range(0, T_all - chunk + 1, stride)) + if starts[-1] != T_all - chunk: + starts.append(T_all - chunk) + return starts + + +def taper_weights(Tc: int, overlap: int, *, is_first: bool, is_last: bool, + device=None, dtype=torch.float32) -> torch.Tensor: + w = torch.ones(Tc, device=device, dtype=dtype) + if overlap <= 0 or Tc <= 1: + return w + n = min(overlap, Tc) + ramp = 0.5 * (1.0 - torch.cos( + math.pi * (torch.arange(n, device=device, dtype=dtype) + 1) / (overlap + 1) + )) + if not is_first: + w[:n] = ramp + if not is_last: + w[Tc - n:] = ramp.flip(0) + return w + + +class ThreadedVideoWriter: + def __init__(self, path: str, fps: float, size: tuple[int, int], + cfg: WriterConfig, *, maxsize: int = 2, channels: int = 3, + color_trc: str | None = None): + self.path = path + self.fps = float(fps) + self.size = size + self.cfg = cfg + self.maxsize = maxsize + self.channels = channels + self.color_trc = color_trc + self.proc = None + self.q = queue.Queue(maxsize=maxsize) + self.err = None + self.thread = threading.Thread(target=self._worker, daemon=True) + self._start() + + def _start(self): + require_ffmpeg(self.cfg.ffmpeg_bin) + w, h = self.size + + in_pix_fmt = "bgra64le" if self.channels == 4 else "bgr48le" + + cmd = [ + self.cfg.ffmpeg_bin, "-y", "-hide_banner", "-loglevel", "error", + "-nostats", "-f", "rawvideo", "-pix_fmt", in_pix_fmt, + "-s:v", f"{w}x{h}", "-r", f"{self.fps:.8f}", "-i", "pipe:0", + "-an" + ] + + if self.channels == 4: + cmd.extend([ + "-c:v", "prores_ks", + "-profile:v", "4", + "-pix_fmt", "yuva444p10le" + ]) + else: + cmd.extend([ + "-c:v", self.cfg.codec, + "-preset", self.cfg.preset, + "-pix_fmt", self.cfg.pix_fmt + ]) + if self.cfg.bitrate: + cmd.extend(["-b:v", self.cfg.bitrate]) + else: + cmd.extend(["-crf", str(int(self.cfg.crf))]) + + trc = self.color_trc if self.color_trc is not None else "iec61966-2-1" + cmd.extend([ + "-threads", str(max(1, self.cfg.threads)), + "-colorspace", "bt709", "-color_primaries", "bt709", + "-color_trc", trc, "-movflags", "+write_colr", + self.path + ]) + + self.proc = subprocess.Popen(cmd, stdin=subprocess.PIPE) + self.thread.start() + + def _worker(self): + try: + while True: + item = self.q.get() + if item is None: + self.q.task_done() + return + self.proc.stdin.write(item.tobytes()) + self.q.task_done() + except Exception as exc: + self.err = exc + + def write(self, frame: np.ndarray): + self.write_many(frame[None]) + + def write_many(self, frames: np.ndarray): + if self.err is not None: + raise self.err + arr = np.ascontiguousarray(frames) + if arr.ndim != 4 or arr.shape[-1] != self.channels: + raise ValueError( + f"expected batched frames as NHWC uint16 with {self.channels} channels, got {arr.shape}") + if arr.dtype != np.uint16: + raise ValueError(f"expected uint16 frames, got {arr.dtype}") + self.q.put(arr) + + def release(self): + self.q.put(None) + self.thread.join() + if self.proc is not None and self.proc.stdin is not None: + self.proc.stdin.close() + rc = self.proc.wait() if self.proc is not None else 0 + if rc != 0 and self.err is None: + self.err = RuntimeError(f"ffmpeg writer failed with exit code {rc}: {self.path}") + if self.err is not None: + raise self.err + + +class ExrSequenceWriter: + """Writes float frames as an OpenEXR image sequence (scene-referred, unclamped). + + Accepts NHWC float32 batches with `channels` channels in BGR(A) order, + matching the layout convention of ThreadedVideoWriter/cv2. Values are + written as-is: no clamping and no quantization. `dtype` selects the EXR + pixel type: "half" (float16, VFX standard) or "float" (full float32). + """ + + def __init__(self, dir_path: str, fps: float, size: tuple[int, int], *, + channels: int = 3, dtype: str = "half", maxsize: int = 2, + frame_name: str = "frame"): + if dtype not in ("half", "float"): + raise ValueError(f"exr dtype must be 'half' or 'float', got {dtype!r}") + self.dir_path = str(dir_path) + self.path = self.dir_path # for logging parity with ThreadedVideoWriter + self.size = size + self.channels = channels + self.frame_name = frame_name + exr_type = (cv2.IMWRITE_EXR_TYPE_HALF if dtype == "half" + else cv2.IMWRITE_EXR_TYPE_FLOAT) + self.clamp_half = dtype == "half" + self.imwrite_params = [ + cv2.IMWRITE_EXR_TYPE, exr_type, + cv2.IMWRITE_EXR_COMPRESSION, cv2.IMWRITE_EXR_COMPRESSION_ZIP, + ] + os.makedirs(self.dir_path, exist_ok=True) + self.frame_idx = 0 + self.q = queue.Queue(maxsize=maxsize) + self.err = None + self.thread = threading.Thread(target=self._worker, daemon=True) + self.thread.start() + + def _worker(self): + try: + while True: + item = self.q.get() + if item is None: + self.q.task_done() + return + idx0, arr = item + if self.clamp_half: + # float16 overflows to inf above 65504; keep half EXRs finite + arr = np.clip(arr, -65504.0, 65504.0) + for j in range(arr.shape[0]): + out_path = os.path.join( + self.dir_path, f"{self.frame_name}_{idx0 + j:06d}.exr") + frame = np.ascontiguousarray(arr[j]) + if frame.shape[-1] == 1: + frame = frame[..., 0] + if not cv2.imwrite(out_path, frame, self.imwrite_params): + raise RuntimeError(f"cv2.imwrite failed: {out_path}") + self.q.task_done() + except Exception as exc: + self.err = exc + + def write_many(self, frames: np.ndarray): + if self.err is not None: + raise self.err + arr = np.ascontiguousarray(frames) + if arr.ndim != 4 or arr.shape[-1] != self.channels: + raise ValueError( + f"expected batched frames as NHWC float32 with " + f"{self.channels} channels, got {arr.shape}") + if arr.dtype != np.float32: + raise ValueError(f"expected float32 frames, got {arr.dtype}") + self.q.put((self.frame_idx, arr)) + self.frame_idx += arr.shape[0] + + def write(self, frame: np.ndarray): + self.write_many(frame[None]) + + def release(self): + self.q.put(None) + self.thread.join() + if self.err is not None: + raise self.err + + +def color_chw_to_bgr(x: np.ndarray) -> np.ndarray: + x = np.clip(np.moveaxis(x, 0, -1), 0, 1) + return cv2.cvtColor((x * 255).astype(np.uint8), cv2.COLOR_RGB2BGR) + + +def gray_1hw_to_bgr(x: np.ndarray) -> np.ndarray: + x = np.clip(x[0], 0, 1) + return cv2.cvtColor((x * 255).astype(np.uint8), cv2.COLOR_GRAY2BGR) + + +def make_checker(H: int, W: int, size: int = 64, + light: float = 0.75, dark: float = 0.5) -> np.ndarray: + y, x = np.indices((H, W)) + mask = (((y // size) + (x // size)) % 2 == 0).astype(np.float32) + gray = mask * light + (1.0 - mask) * dark + return np.stack([gray, gray, gray], axis=0).astype(np.float32) + + +def composite_over_checker_srgb(fg_srgb_chw: np.ndarray, alpha_1hw: np.ndarray, + checker_chw: np.ndarray) -> np.ndarray: + fg_lin = cu.srgb_to_linear(fg_srgb_chw) + bg_lin = cu.srgb_to_linear(checker_chw) + alpha = np.clip(alpha_1hw, 0.0, 1.0) + comp_lin = fg_lin * alpha + bg_lin * (1.0 - alpha) + return np.clip(cu.linear_to_srgb(np.maximum(comp_lin, 0.0)), 0.0, 1.0) + + +def corridorkey_despill_average_srgb(fg_srgb_chw: np.ndarray, + strength: float = 0.5) -> np.ndarray: + if strength <= 0.0: + return fg_srgb_chw + r = fg_srgb_chw[0] + g = fg_srgb_chw[1] + b = fg_srgb_chw[2] + spill = np.maximum(g - (r + b) * 0.5, 0.0) + full = np.stack([r + spill * 0.5, g - spill, b + spill * 0.5], axis=0) + if strength >= 1.0: + return full.astype(np.float32, copy=False) + return fg_srgb_chw * (1.0 - strength) + full * strength diff --git a/requirements.txt b/requirements.txt index f688347d6c718fc3ff66bb42ea8e63dab9ec3f4b..26fc6e9539ff1008722916ac231c371a3576b3d5 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,3 +10,10 @@ torch torchvision timm https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.0/flash_attn-2.8.3+cu126torch2.9-cp310-cp310-linux_x86_64.whl +transformers>=4.57,<5 +einops>=0.8 +scipy +safetensors>=0.6 +pillow +tqdm +utils3d @ git+https://github.com/EasternJournalist/utils3d.git@83bfa5bda008803f6521ce9547f94023b8afeddd diff --git a/third_party/MoGe/LICENSE b/third_party/MoGe/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..3458b5ccd398afed340e17a4d0615c9a8666bb5d --- /dev/null +++ b/third_party/MoGe/LICENSE @@ -0,0 +1,224 @@ + MIT License + + Copyright (c) Microsoft Corporation. + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE + + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/third_party/MoGe/README.md b/third_party/MoGe/README.md new file mode 100644 index 0000000000000000000000000000000000000000..3e5996e8b97f9f198092c67d349830a35465df5b --- /dev/null +++ b/third_party/MoGe/README.md @@ -0,0 +1,295 @@ +# MoGe: Accurate Monocular Geometry Estimation + +MoGe is a powerful model for recovering 3D geometry from monocular open-domain images, including metric point maps, metric depth maps, normal maps and camera FOV. ***Check our websites ([MoGe-1](https://wangrc.site/MoGePage), [MoGe-2](https://wangrc.site/MoGe2Page)) for videos and interactive results!*** + +## 📖 Publications + +### MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details + +
+ +### MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision + + + +
+
+
+## 🌟 Features
+
+* **Accurate 3D geometry estimation**: Estimate point maps & depth maps & [normal maps](docs/normal.md) from open-domain single images with high precision -- all capabilities in one model, one forward pass.
+* **Optional ground-truth FOV input**: Enhance model accuracy further by providing the true field of view.
+* **Flexible resolution support**: Works seamlessly with various resolutions and aspect ratios, from 2:1 to 1:2.
+* **Optimized for speed**: Achieves 60ms latency per image (A100 or RTX3090, FP16, ViT-L). Adjustable inference resolution for even faster speed.
+
+## ✨ News
+
+***(2025-10-16)***
+* Updated training code for MoGe-2.
+
+***(2025-06-10)***
+
+* ❗**Released MoGe-2**, a state-of-the-art model for monocular geometry, with these new capabilities in one unified model:
+ * point map prediction in **metric scale**;
+ * comparable and even better performance over MoGe-1;
+ * significant improvement of **visual sharpness**;
+ * high-quality [**normal map** estimation](docs/normal.md);
+ * lower inference latency.
+
+## 📦 Installation
+
+### Install via pip
+
+```bash
+pip install git+https://github.com/microsoft/MoGe.git
+```
+
+### Or clone this repository
+
+```bash
+git clone https://github.com/microsoft/MoGe.git
+cd MoGe
+pip install -r requirements.txt # install the requirements
+```
+
+Note: MoGe should be compatible with most requirements versions. Please check the `requirements.txt` for more details if you encounter any dependency issues.
+
+## 🤗 Pretrained Models
+
+Our pretrained models are available on the huggingface hub:
+
+| Version | +Hugging Face Model | +Metric scale | +Normal | +#Params | +
|---|---|---|---|---|
| MoGe-1 | +Ruicheng/moge-vitl |
+ - | +- | +314M | +
| MoGe-2 | +Ruicheng/moge-2-vitl |
+ ✅ | +- | +326M | +
Ruicheng/moge-2-vitl-normal |
+ ✅ | +✅ | +331M | +|
Ruicheng/moge-2-vitb-normal |
+ ✅ | +✅ | +104M | +|
Ruicheng/moge-2-vits-normal |
+ ✅ | +✅ | +35M | +
+
+The photo is from [this URL](https://commons.wikimedia.org/wiki/Category:360%C2%B0_panoramas_with_equirectangular_projection#/media/File:Braunschweig_Sankt-%C3%84gidien_Panorama_02.jpg)
+