DollasAndSpence GuardSkill commited on
Commit
89ae121
·
0 Parent(s):

Duplicate from joyfox/LTX2.3-ICEdit-Insight

Browse files

Co-authored-by: GuardSkill <GuardSkill@users.noreply.huggingface.co>

.gitattributes ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ assets/effects/output_004.webp filter=lfs diff=lfs merge=lfs -text
37
+ assets/effects/output_005.webp filter=lfs diff=lfs merge=lfs -text
38
+ assets/effects/output_004_bigger.webp filter=lfs diff=lfs merge=lfs -text
39
+ assets/effects/视频高清对比效果.webp filter=lfs diff=lfs merge=lfs -text
40
+ assets/effects/去水印对比效果.webp filter=lfs diff=lfs merge=lfs -text
41
+ assets/effects/去字幕对比效果.webp filter=lfs diff=lfs merge=lfs -text
42
+ assets/effects/视频修复对比效果2.webp filter=lfs diff=lfs merge=lfs -text
43
+ assets/effects/视频高清对比效果2.webp filter=lfs diff=lfs merge=lfs -text
44
+ assets/effects/去水印对比效果2.webp filter=lfs diff=lfs merge=lfs -text
45
+ assets/effects/去字幕对比效果2.webp filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction, and
10
+ distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by the copyright
13
+ owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all other entities
16
+ that control, are controlled by, or are under common control with that entity.
17
+ For the purposes of this definition, "control" means (i) the power, direct or
18
+ indirect, to cause the direction or management of such entity, whether by
19
+ contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the
20
+ outstanding shares, or (iii) beneficial ownership of such entity.
21
+
22
+ "You" (or "Your") shall mean an individual or Legal Entity exercising
23
+ permissions granted by this License.
24
+
25
+ "Source" form shall mean the preferred form for making modifications, including
26
+ but not limited to software source code, documentation source, and configuration
27
+ files.
28
+
29
+ "Object" form shall mean any form resulting from mechanical transformation or
30
+ translation of a Source form, including but not limited to compiled object code,
31
+ generated documentation, and conversions to other media types.
32
+
33
+ "Work" shall mean the work of authorship, whether in Source or Object form, made
34
+ available under the License, as indicated by a copyright notice that is included
35
+ in or attached to the work (an example is provided in the Appendix below).
36
+
37
+ "Derivative Works" shall mean any work, whether in Source or Object form, that
38
+ is based on (or derived from) the Work and for which the editorial revisions,
39
+ annotations, elaborations, or other modifications represent, as a whole, an
40
+ original work of authorship. For the purposes of this License, Derivative Works
41
+ shall not include works that remain separable from, or merely link (or bind by
42
+ name) to the interfaces of, the Work and Derivative Works thereof.
43
+
44
+ "Contribution" shall mean any work of authorship, including the original version
45
+ of the Work and any modifications or additions to that Work or Derivative Works
46
+ thereof, that is intentionally submitted to Licensor for inclusion in the Work
47
+ by the copyright owner or by an individual or Legal Entity authorized to submit
48
+ on behalf of the copyright owner. For the purposes of this definition,
49
+ "submitted" means any form of electronic, verbal, or written communication sent
50
+ to the Licensor or its representatives, including but not limited to
51
+ communication on electronic mailing lists, source code control systems, and
52
+ issue tracking systems that are managed by, or on behalf of, the Licensor for
53
+ the purpose of discussing and improving the Work, but excluding communication
54
+ that is conspicuously marked or otherwise designated in writing by the copyright
55
+ owner as "Not a Contribution."
56
+
57
+ "Contributor" shall mean Licensor and any individual or Legal Entity on behalf
58
+ of whom a Contribution has been received by Licensor and subsequently
59
+ incorporated within the Work.
60
+
61
+ 2. Grant of Copyright License. Subject to the terms and conditions of this
62
+ License, each Contributor hereby grants to You a perpetual, worldwide,
63
+ non-exclusive, no-charge, royalty-free, irrevocable copyright license to
64
+ reproduce, prepare Derivative Works of, publicly display, publicly perform,
65
+ sublicense, and distribute the Work and such Derivative Works in Source or
66
+ Object form.
67
+
68
+ 3. Grant of Patent License. Subject to the terms and conditions of this License,
69
+ each Contributor hereby grants to You a perpetual, worldwide, non-exclusive,
70
+ no-charge, royalty-free, irrevocable (except as stated in this section) patent
71
+ license to make, have made, use, offer to sell, sell, import, and otherwise
72
+ transfer the Work, where such license applies only to those patent claims
73
+ licensable by such Contributor that are necessarily infringed by their
74
+ Contribution(s) alone or by combination of their Contribution(s) with the Work
75
+ to which such Contribution(s) was submitted. If You institute patent litigation
76
+ against any entity (including a cross-claim or counterclaim in a lawsuit)
77
+ alleging that the Work or a Contribution incorporated within the Work
78
+ constitutes direct or contributory patent infringement, then any patent licenses
79
+ granted to You under this License for that Work shall terminate as of the date
80
+ such litigation is filed.
81
+
82
+ 4. Redistribution. You may reproduce and distribute copies of the Work or
83
+ Derivative Works thereof in any medium, with or without modifications, and in
84
+ Source or Object form, provided that You meet the following conditions:
85
+
86
+ (a) You must give any other recipients of the Work or Derivative Works a copy of
87
+ this License; and
88
+
89
+ (b) You must cause any modified files to carry prominent notices stating that
90
+ You changed the files; and
91
+
92
+ (c) You must retain, in the Source form of any Derivative Works that You
93
+ distribute, all copyright, patent, trademark, and attribution notices from the
94
+ Source form of the Work, excluding those notices that do not pertain to any part
95
+ of the Derivative Works; and
96
+
97
+ (d) If the Work includes a "NOTICE" text file as part of its distribution, then
98
+ any Derivative Works that You distribute must include a readable copy of the
99
+ attribution notices contained within such NOTICE file, excluding those notices
100
+ that do not pertain to any part of the Derivative Works, in at least one of the
101
+ following places: within a NOTICE text file distributed as part of the
102
+ Derivative Works; within the Source form or documentation, if provided along
103
+ with the Derivative Works; or, within a display generated by the Derivative
104
+ Works, if and wherever such third-party notices normally appear. The contents of
105
+ the NOTICE file are for informational purposes only and do not modify the
106
+ License. You may add Your own attribution notices within Derivative Works that
107
+ You distribute, alongside or as an addendum to the NOTICE text from the Work,
108
+ provided that such additional attribution notices cannot be construed as
109
+ modifying the License.
110
+
111
+ You may add Your own copyright statement to Your modifications and may provide
112
+ additional or different license terms and conditions for use, reproduction, or
113
+ distribution of Your modifications, or for any such Derivative Works as a whole,
114
+ provided Your use, reproduction, and distribution of the Work otherwise complies
115
+ with the conditions stated in this License.
116
+
117
+ 5. Submission of Contributions. Unless You explicitly state otherwise, any
118
+ Contribution intentionally submitted for inclusion in the Work by You to the
119
+ Licensor shall be under the terms and conditions of this License, without any
120
+ additional terms or conditions. Notwithstanding the above, nothing herein shall
121
+ supersede or modify the terms of any separate license agreement you may have
122
+ executed with Licensor regarding such Contributions.
123
+
124
+ 6. Trademarks. This License does not grant permission to use the trade names,
125
+ trademarks, service marks, or product names of the Licensor, except as required
126
+ for reasonable and customary use in describing the origin of the Work and
127
+ reproducing the content of the NOTICE file.
128
+
129
+ 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in
130
+ writing, Licensor provides the Work (and each Contributor provides its
131
+ Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
132
+ KIND, either express or implied, including, without limitation, any warranties
133
+ or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
134
+ PARTICULAR PURPOSE. You are solely responsible for determining the
135
+ appropriateness of using or redistributing the Work and assume any risks
136
+ associated with Your exercise of permissions under this License.
137
+
138
+ 8. Limitation of Liability. In no event and under no legal theory, whether in
139
+ tort (including negligence), contract, or otherwise, unless required by
140
+ applicable law (such as deliberate and grossly negligent acts) or agreed to in
141
+ writing, shall any Contributor be liable to You for damages, including any
142
+ direct, indirect, special, incidental, or consequential damages of any character
143
+ arising as a result of this License or out of the use or inability to use the
144
+ Work (including but not limited to damages for loss of goodwill, work stoppage,
145
+ computer failure or malfunction, or any and all other commercial damages or
146
+ losses), even if such Contributor has been advised of the possibility of such
147
+ damages.
148
+
149
+ 9. Accepting Warranty or Additional Liability. While redistributing the Work or
150
+ Derivative Works thereof, You may choose to offer, and charge a fee for,
151
+ acceptance of support, warranty, indemnity, or other liability obligations
152
+ and/or rights consistent with this License. However, in accepting such
153
+ obligations, You may act only on Your own behalf and on Your sole responsibility,
154
+ not on behalf of any other Contributor, and only if You agree to indemnify,
155
+ defend, and hold each Contributor harmless for any liability incurred by, or
156
+ claims asserted against, such Contributor by reason of your accepting any such
157
+ warranty or additional liability.
158
+
159
+ END OF TERMS AND CONDITIONS
README.md ADDED
@@ -0,0 +1,330 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ library_name: diffusers
6
+ base_model:
7
+ - Lightricks/LTX-2.3
8
+ pipeline_tag: video-to-video
9
+ tags:
10
+ - video-editing
11
+ - video-restoration
12
+ - ltx-video
13
+ - ltx-2-3
14
+ - dit
15
+ - ic-lora
16
+ - watermark-removal
17
+ - subtitle-removal
18
+ - super-resolution
19
+ - hd-enhancement
20
+ - joyfox
21
+ ---
22
+
23
+ # LTX2.3-ICEdit-Insight
24
+
25
+ <table>
26
+ <tr>
27
+ <td align="center"><img src="./assets/effects/output_004.webp" alt="Video restoration preview" width="420"/></td>
28
+ <td align="center"><img src="./assets/effects/视频高清对比效果.webp" alt="Video HD enhancement preview" width="420"/></td>
29
+ </tr>
30
+ <tr>
31
+ <td align="center"><img src="./assets/effects/去水印对比效果.webp" alt="Watermark removal preview" width="420"/></td>
32
+ <td align="center"><img src="./assets/effects/去字幕对比效果.webp" alt="Subtitle removal preview" width="420"/></td>
33
+ </tr>
34
+ </table>
35
+
36
+ **LTX2.3-ICEdit-Insight** is a task-aware video restoration and editing model family developed by **JoyFox Lab**, built on top of the **LTX-2.3 DiT-based audio-video foundation model**.
37
+
38
+ This release focuses on four practical video editing directions:
39
+
40
+ - **Video Restoration**: degradation recovery, compression cleanup, blur and noise reduction, and damaged detail restoration.
41
+ - **Video HD Enhancement**: super-resolution, detail reconstruction, texture sharpening, and perceptual quality improvement.
42
+ - **Watermark Removal**: logo cleanup, semi-transparent overlay removal, and occlusion-aware background reconstruction.
43
+ - **Subtitle Removal**: hard subtitle removal, caption cleanup, text overlay removal, and temporally stable inpainting.
44
+
45
+ Unlike conventional frame-level enhancement pipelines, this model family operates as a **generative video restoration system** in latent video space. It is designed to preserve global structure, camera motion, object identity, and temporal consistency while reconstructing missing or degraded visual content.
46
+
47
+ Project links: [GitHub project](https://github.com/Valiant-Cat/LTX2-ICEdit-Insight) | [JoyFox on Hugging Face](https://huggingface.co/joyfox) | [Paper (Research Square)](https://www.researchsquare.com/article/rs-9775063/v1) | [DOI](https://doi.org/10.21203/rs.3.rs-9775063/v1)
48
+
49
+ ## 📄 Paper
50
+
51
+ - **Title**: *LTX-Insight: Unified Video Restoration and Semantic Editing via Task-Aware Adaptation and Temporal Consistency*
52
+ - **Authors**: Fan Tang, Siyuan Li
53
+ - **Venue**: Research Square
54
+ - **Posted date**: May 25, 2026
55
+ - **DOI**: [`10.21203/rs.3.rs-9775063/v1`](https://doi.org/10.21203/rs.3.rs-9775063/v1)
56
+ - **Paper URL**: https://www.researchsquare.com/article/rs-9775063/v1
57
+
58
+ This model release corresponds to the paper's unified video post-processing framework built around three high-level settings: video super-resolution/enhancement, occlusion removal and repair, and instruction-driven semantic editing.
59
+
60
+ In the currently released inference package, those ideas are exposed through four practical routes:
61
+
62
+ - **Video Restoration**
63
+ - **Video HD Enhancement**
64
+ - **Watermark Removal**
65
+ - **Subtitle Removal**
66
+
67
+ The paper introduces three core components:
68
+
69
+ - **TADA (Task-Aware Decoupled Adapter)** for separating structure restoration from texture/semantic editing under a shared backbone.
70
+ - **FGLC (Flow-Guided Latent Consistency)** for explicit temporal consistency modeling in latent space using optical-flow alignment and cross-frame attention bias.
71
+ - **PMTR (Progressive Multi-Task Routing)** for more stable joint multi-task training through curriculum scheduling and collaborative routing.
72
+
73
+ ## 📦 Model Files
74
+
75
+ | File | Purpose |
76
+ | --- | --- |
77
+ | `ltx-2.3-edit-insight-dev-fp8.safetensors` | Unified Insight base checkpoint for LTX-2.3 editing |
78
+ | `ltx2.3-video-restoration-general.safetensors` | Video restoration, artifact cleanup, blur and noise recovery |
79
+ | `ltx2.3-ic-video-upscale-general.safetensors` | Video HD enhancement, super-resolution, and detail recovery |
80
+ | `ltx2.3-ic-watermark-remove-general.safetensors` | Watermark removal and occlusion-aware reconstruction |
81
+ | `ltx2.3-ic-subtitles-remove-general.safetensors` | Subtitle removal and text overlay cleanup |
82
+
83
+ ## 🎬 Showcase
84
+
85
+ <table>
86
+ <tr>
87
+ <td align="center"><b>Video Restoration</b></td>
88
+ <td align="center"><b>Video HD Enhancement</b></td>
89
+ </tr>
90
+ <tr>
91
+ <td align="center"><img src="./assets/effects/output_004.webp" alt="Video restoration preview" width="600"/></td>
92
+ <td align="center"><img src="./assets/effects/视频高清对比效果.webp" alt="Video HD enhancement preview" width="600"/></td>
93
+ </tr>
94
+ <tr>
95
+ <td align="center"><b>Watermark Removal</b></td>
96
+ <td align="center"><b>Subtitle Removal</b></td>
97
+ </tr>
98
+ <tr>
99
+ <td align="center"><img src="./assets/effects/去水印对比效果.webp" alt="Watermark removal preview" width="600"/></td>
100
+ <td align="center"><img src="./assets/effects/去字幕对比效果.webp" alt="Subtitle removal preview" width="600"/></td>
101
+ </tr>
102
+ </table>
103
+
104
+ <br/>
105
+
106
+ <table>
107
+ <tr>
108
+ <td align="center"><b>Video Restoration</b></td>
109
+ <td align="center"><b>Video HD Enhancement</b></td>
110
+ </tr>
111
+ <tr>
112
+ <td align="center"><img src="./assets/effects/视频修复对比效果2.webp" alt="Video restoration preview 2" width="600"/></td>
113
+ <td align="center"><img src="./assets/effects/视频高清对比效果2.webp" alt="Video HD enhancement preview 2" width="600"/></td>
114
+ </tr>
115
+ <tr>
116
+ <td align="center"><b>Watermark Removal</b></td>
117
+ <td align="center"><b>Subtitle Removal</b></td>
118
+ </tr>
119
+ <tr>
120
+ <td align="center"><img src="./assets/effects/去水印对比效果2.webp" alt="Watermark removal preview 2" width="600"/></td>
121
+ <td align="center"><img src="./assets/effects/去字幕对比效果2.webp" alt="Subtitle removal preview 2" width="600"/></td>
122
+ </tr>
123
+ </table>
124
+
125
+ ## 🚀 Script Usage
126
+
127
+ Run all scripts from the project root.
128
+
129
+ ```bash
130
+ bash run_restoration.sh
131
+ bash run_hd.sh
132
+ bash run_hd.sh /path/to/input.mp4
133
+ bash run_watermark_rm.sh
134
+ bash run_watermark_rm.sh /path/to/input.mp4
135
+ bash run_subtitle_rm.sh
136
+ bash run_subtitle_rm.sh /path/to/input.mp4
137
+ ```
138
+
139
+ ## 💻 Command Examples
140
+
141
+ ### Video Restoration
142
+
143
+ ```bash
144
+ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
145
+ python run_pipeline.py \
146
+ --mode restoration \
147
+ --video ./inputs/input_480p.mp4 \
148
+ --prompt "Convert the video to ultra-high-definition quality while removing artifacts and rebuilding high-frequency details." \
149
+ --output ./outputs/output_restoration.mp4 \
150
+ --height 1184 --width 704 --num-frames 97 \
151
+ --fps 24.0 --seed 42 \
152
+ --sigma-profile workflow \
153
+ --streaming-prefetch-count 2 \
154
+ --model-checkpoint ./models/checkpoints/ltx-2.3-edit-insight-dev-fp8.safetensors \
155
+ --lora ./models/loras/ltx2.3-train/ltx2.3-video-restoration-general.safetensors
156
+ ```
157
+
158
+ ### Video HD Enhancement
159
+
160
+ ```bash
161
+ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
162
+ python run_pipeline.py \
163
+ --mode hd \
164
+ --video ./inputs/input_480p.mp4 \
165
+ --prompt "Convert the video to ultra-high-definition quality, significantly improving clarity, fine detail richness, texture fidelity, and overall perceptual sharpness." \
166
+ --output ./outputs/output_hd.mp4 \
167
+ --height 1184 --width 704 --num-frames 97 \
168
+ --fps 24.0 --seed 42 \
169
+ --sigma-profile workflow \
170
+ --streaming-prefetch-count 2 \
171
+ --model-checkpoint ./models/checkpoints/ltx-2.3-edit-insight-dev-fp8.safetensors \
172
+ --lora ./models/loras/ltx2.3-train/ltx2.3-ic-video-upscale-general.safetensors
173
+ ```
174
+
175
+ ### Watermark Removal
176
+
177
+ ```bash
178
+ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
179
+ python run_pipeline.py \
180
+ --mode watermark_rm \
181
+ --video ./inputs/input_480p.mp4 \
182
+ --prompt "Remove short-video platform watermarks and related occlusions from the video, restoring a clean, clear, and natural original image." \
183
+ --output ./outputs/output_watermark_rm.mp4 \
184
+ --height 1184 --width 704 --num-frames 97 \
185
+ --fps 24.0 --seed 1546 \
186
+ --sigma-profile workflow \
187
+ --streaming-prefetch-count 2 \
188
+ --model-checkpoint ./models/checkpoints/ltx-2.3-edit-insight-dev-fp8.safetensors \
189
+ --lora ./models/loras/ltx2.3-train/ltx2.3-ic-watermark-remove-general.safetensors
190
+ ```
191
+
192
+ ### Subtitle Removal
193
+
194
+ ```bash
195
+ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
196
+ python run_pipeline.py \
197
+ --mode subtitle_rm \
198
+ --video ./inputs/input_480p.mp4 \
199
+ --prompt "Remove subtitles, captions, and related text occlusions from the video, restoring a clean and natural underlying image." \
200
+ --output ./outputs/output_subtitle_rm.mp4 \
201
+ --height 1184 --width 704 --num-frames 97 \
202
+ --fps 24.0 --seed 42 \
203
+ --sigma-profile workflow \
204
+ --streaming-prefetch-count 2 \
205
+ --model-checkpoint ./models/checkpoints/ltx-2.3-edit-insight-dev-fp8.safetensors \
206
+ --lora ./models/loras/ltx2.3-train/ltx2.3-ic-subtitles-remove-general.safetensors
207
+ ```
208
+
209
+ ## ✨ Key Improvements
210
+
211
+ ### Task-Aware IC-Edit Framework
212
+
213
+ We introduce a task-aware IC-Edit training framework for LTX-2.3, where each restoration direction is optimized with dedicated instruction conditioning and task-specific IC-LoRA adapters.
214
+
215
+ The model is trained not only to improve visual quality, but also to understand the editing goal behind different restoration tasks, including watermark removal, subtitle cleanup, damaged region recovery, and high-definition enhancement.
216
+
217
+ ### LTX-2.3 DiT Backbone Adaptation
218
+
219
+ The model family is built on the LTX-2.3 foundation architecture, a diffusion-transformer video model designed for high-fidelity image-to-video and video generation workflows.
220
+
221
+ Our adaptation targets video restoration by improving:
222
+
223
+ - latent-space editability
224
+ - instruction-following behavior
225
+ - frame-to-frame stability
226
+ - high-frequency detail recovery
227
+ - local reconstruction around degraded or occluded regions
228
+
229
+ ### Spatiotemporal Consistency Optimization
230
+
231
+ Video restoration requires more than strong single-frame quality. We optimize temporal consistency so that restored areas remain stable across adjacent frames.
232
+
233
+ This reduces common artifacts such as:
234
+
235
+ - flickering textures
236
+ - unstable reconstructed backgrounds
237
+ - inconsistent watermark removal
238
+ - subtitle ghosting
239
+ - frame-wise color shift
240
+ - detail popping during motion
241
+
242
+ ### Degradation-Aware Training Curriculum
243
+
244
+ The training curriculum covers realistic video defects including:
245
+
246
+ - compression artifacts
247
+ - motion blur
248
+ - sensor noise
249
+ - low-bitrate video
250
+ - text overlays
251
+ - hard subtitles
252
+ - semi-transparent watermarks
253
+ - platform logos
254
+ - local occlusions
255
+ - low-resolution inputs
256
+
257
+ This improves generalization across short videos, social-media clips, mobile footage, downloaded videos, and compressed production material.
258
+
259
+ ### Occlusion-Aware Reconstruction
260
+
261
+ For watermark and subtitle removal, the model is optimized to reconstruct the hidden visual content behind occluded regions.
262
+
263
+ Instead of smearing or blurring the target area, it uses surrounding spatial context and temporal cues to infer plausible background structure, object boundaries, lighting, and texture continuity.
264
+
265
+ ### Frequency-Enhanced HD Restoration
266
+
267
+ For HD enhancement, the model improves perceptual sharpness and fine visual detail through frequency-aware restoration training.
268
+
269
+ This is especially helpful for recovering:
270
+
271
+ - hair strands
272
+ - fabric texture
273
+ - skin detail
274
+ - product edges
275
+ - background patterns
276
+ - typography-like fine structures
277
+ - natural image clarity
278
+
279
+ ## 🧠 Inference Notes
280
+
281
+ - Single-stage inference is recommended for most editing tasks.
282
+ - Two-stage refinement can improve visual polish but may weaken task-specific LoRA constraints.
283
+ - Watermark and subtitle removal perform best when the occlusion area is stable and not excessively large.
284
+ - HD enhancement quality depends on input resolution, motion complexity, and compression level.
285
+ - Higher output resolution improves detail but requires more VRAM.
286
+ - For strong-motion videos, conservative denoising settings are recommended to preserve temporal structure.
287
+ - Frame count should follow the `8k + 1` rule.
288
+ - Output height and width should be multiples of `32` in single-stage inference.
289
+
290
+ ## 📚 Citation
291
+
292
+ If this model family is useful in your research or application workflow, please cite:
293
+
294
+ ```bibtex
295
+ @article{tang2026ltxinsight,
296
+ title = {LTX-Insight: Unified Video Restoration and Semantic Editing via Task-Aware Adaptation and Temporal Consistency},
297
+ author = {Tang, Fan and Li, Siyuan},
298
+ journal = {Research Square},
299
+ year = {2026},
300
+ month = {05},
301
+ doi = {10.21203/rs.3.rs-9775063/v1},
302
+ url = {https://www.researchsquare.com/article/rs-9775063/v1}
303
+ }
304
+ ```
305
+
306
+ ## 🏗️ Training
307
+
308
+ This model family was trained and optimized by **JoyFox Lab** (**Chengdu Xuanhu Technology Co., Ltd.**).
309
+
310
+ The training pipeline includes:
311
+
312
+ - task-aware video restoration data construction
313
+ - degradation synthesis and curriculum training
314
+ - IC-LoRA specialization for four editing directions
315
+ - temporal consistency regularization
316
+ - occlusion-aware reconstruction training
317
+ - high-frequency perceptual enhancement
318
+ - instruction-guided video editing optimization
319
+
320
+ ## 📬 Contact
321
+
322
+ For research collaboration, commercial licensing, or workflow integration, contact:
323
+
324
+ - `z@vvicat.com`
325
+
326
+ ## 📜 License
327
+
328
+ Licensed under **Apache 2.0**.
329
+
330
+ Please also review the license terms of the upstream LTX-2.3 base model when using or redistributing derivative checkpoints.
assets/effects/output_004.webp ADDED

Git LFS Details

  • SHA256: 65302900da6e3ddcb6ee13f08c2888149e13dfce93be14a48dca63fa3756469e
  • Pointer size: 131 Bytes
  • Size of remote file: 939 kB
assets/effects/output_004_bigger.webp ADDED

Git LFS Details

  • SHA256: b3419082f0da2571ef82b2e1ed2ab9d2ad15584515f8d30277b52a9f5c68df70
  • Pointer size: 133 Bytes
  • Size of remote file: 28.6 MB
assets/effects/output_005.webp ADDED

Git LFS Details

  • SHA256: 944828a9b963caed299860b4401471bff331187c1cab1bcd0a0201659658eef4
  • Pointer size: 131 Bytes
  • Size of remote file: 745 kB
assets/effects/去字幕对比效果.webp ADDED

Git LFS Details

  • SHA256: bc2b9d28804f0feb172d7b4a3a8b9c6c155770bb267017131ec5996442fa6eb5
  • Pointer size: 132 Bytes
  • Size of remote file: 1.2 MB
assets/effects/去字幕对比效果2.webp ADDED

Git LFS Details

  • SHA256: e4d1ea03cfc4aa9bb1a57fc615888e1a8cbfa6112d0b3695c7cb15b7c73a6320
  • Pointer size: 132 Bytes
  • Size of remote file: 6.44 MB
assets/effects/去水印对比效果.webp ADDED

Git LFS Details

  • SHA256: 4579f356dfd0146188a188728382a81b645dc51a2b12b0d4889c4c222a9060a3
  • Pointer size: 131 Bytes
  • Size of remote file: 887 kB
assets/effects/去水印对比效果2.webp ADDED

Git LFS Details

  • SHA256: 9d7911e55cc41f8cc367e818896bd1f98f3bc023b8147f9c19cb93f5b2837f05
  • Pointer size: 131 Bytes
  • Size of remote file: 983 kB
assets/effects/视频修复对比效果2.webp ADDED

Git LFS Details

  • SHA256: f82c8a2e0b4c5be9cded7f5096b5b18bd507b992bd981785e9efddc249c64cc8
  • Pointer size: 131 Bytes
  • Size of remote file: 603 kB
assets/effects/视频高清对比效果.webp ADDED

Git LFS Details

  • SHA256: a3ddb03f14f06f4f1b262d260d813375c1a11f4308930900d9f73fd7f691734c
  • Pointer size: 132 Bytes
  • Size of remote file: 2.83 MB
assets/effects/视频高清对比效果2.webp ADDED

Git LFS Details

  • SHA256: 9714f6c1912d960997fff63b960ccaeb063805a604e39aa21d79901df61fdc44
  • Pointer size: 132 Bytes
  • Size of remote file: 2.03 MB
ltx-2.3-edit-insight-dev-fp8.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b729383403480fa8faabbbe3d8aa78109e0d5ecc15d80e3de5dc51fdcbc755bf
3
+ size 29135982126
ltx2.3-ic-subtitles-remove-general.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b3d73d7bf737dcfeba6bfa310105532f8f34aa23a3f9e01cb9f873341b8ef88f
3
+ size 327287384
ltx2.3-ic-video-upscale-general.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4cee140e1d71fc67ea8b874b1c7892b0a9ce92aba9bb3fe6e877ba47ac3ab762
3
+ size 327287384
ltx2.3-ic-watermark-remove-general.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f935996f9725a452a45b5e553b8cc0142b64c11d332792f5866295adc5aa7fb1
3
+ size 327287384
ltx2.3-video-restoration-general.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0460ace57d2ca74356978abc0fb3302b1f20b4deb6961cdb348e77232f84dfb3
3
+ size 100767488