Using Stable Diffision models for Colorization

[Dan64]

Hi safshe, 

    LTX 2.3 has  native limitations:
 

1.  Maximum duration per single clip: The model natively generates segments up to 20 seconds per pass 
   
2.  Temporal coherence limitations: Generating 10 continuous minutes or more would require a large amount of video memory (VRAM) and temporal context stability that the current architecture of Diffusion Transformers (DiT) cannot handle without completely losing the thread of the plot, physics, and characters after a few dozen seconds.
   

   So I don't think that LTX is currently a viable solution for the coloring movies with consistent colorization. 
   Indeed I think that HAVCServerDiT is a practical working solution

   I made a test with a 90min clip. I generated about 3500 reference images, to introduce color consistency I removed the inconsistent frames and changed the colors of the necessary frames with "Fix Image" and/or "Fix Colors". In about 4h I was able to obtain all the necessary reference frames with the correct colors. Previously the video colorization was an activity that required several months of manual colorization activity. Non you can obtain about the same quality in 8h (4h for fix reference frames + 4h for encoding).
  I think that is a significant improvement.

Dan

[safshe]

Hi safshe, 

    LTX 2.3 has  native limitations:
 

1.  Maximum duration per single clip: The model natively generates segments up to 20 seconds per pass 
   
2.  Temporal coherence limitations: Generating 10 continuous minutes or more would require a large amount of video memory (VRAM) and temporal context stability that the current architecture of Diffusion Transformers (DiT) cannot handle without completely losing the thread of the plot, physics, and characters after a few dozen seconds.
   

   So I don't think that LTX is currently a viable solution for the coloring movies with consistent colorization. 
   Indeed I think that HAVCServerDiT is a practical working solution

   I made a test with a 90min clip. I generated about 3500 reference images, to introduce color consistency I removed the inconsistent frames and changed the colors of the necessary frames with "Fix Image" and/or "Fix Colors". In about 4h I was able to obtain all the necessary reference frames with the correct colors. Previously the video colorization was an activity that required several months of manual colorization activity. Non you can obtain about the same quality in 8h (4h for fix reference frames + 4h for encoding).
  I think that is a significant improvement.

Dan

I tried LTX 2.3. Tested a 37 second clip. The color consistency was great but as you said long clip need more Vram. The best practice is to work shot by shot.

[didris]

I added the new Tab "Fix Colors" to the project HAVCServerDiT



Using this tab it is possible to propagate the colors from the Reference Image (on the left) to the Target Image (in the middle). The resulting colorized image is shown on the right.

The best model to perform this color propagation is CMNET2 which is the model used for this colorization. In case the color propagation is not satisfactory, it is possible to use the button "Copy -> Fix Image" and the colorized image is provided as input to the Tab "Fix Image" where it will be possible to use Qwen-Image-Edit to apply additional color adjustments.

Dan

P.S.
Please update the code as described in the README

Hi, Dan

I wanted to share a suggestion regarding the new “Fix Image” feature — it’s really impressive and a strong step forward for improving color consistency.
One possible improvement could be adding support for selecting multiple target images at once when applying a reference image. At the moment, each image needs to be processed individually, which can be time-consuming in workflows involving sequences or multiple frames.

In many cases, scenes consist of several related images (for example 5–6 frames) that need to share the same color treatment. Allowing batch selection of target images and applying a single reference-based recolor operation to all of them would make the workflow significantly faster and more efficient.

Overall, great feature — this addition could make it even more powerful for maintaining consistency across image sets.

[Dan64]

I wanted to share a suggestion regarding the new “Fix Image” feature — it’s really impressive and a strong step forward for improving color consistency.

Not clear if you are asking the batch fix, for the Tab "Fix Image" or "Fix Colors".
Could you clarify ?

Dan

[didris]

I wanted to share a suggestion regarding the new “Fix Image” feature — it’s really impressive and a strong step forward for improving color consistency.

Not clear if you are asking the batch fix, for the Tab "Fix Image" or "Fix Colors".
Could you clarify ?

Dan

Hi, Dan
batch fix for tab  Fix Colors, but if possible and for Fix Image

Best regards

[Dan64]

Hi didris

  I just added the "Batch Processing"  to "Fix Image" & "Fix Colors", see README

  But I don't understand your need. CMNET2 is very good in propagating the colors, while with Colormnet it was necessary to add a lot of reference frames, with CMNET2 are necessary few reference frames. A reference frame every 30 or every 50 frames should be enough. Increasing the number of reference frames you are not contributing to improve the colorization, but on the contrary you risk obtaining the opposite effect of worsening the overall quality due to the risk of increasing the colors inconsistency.

Dan

P.S.
see also this issue: #5

[hallomanbh]

One suggestion that I think could significantly improve frame-to-frame color consistency is to allow manual color specification for different objects within a shot.
For example, if I specify:

• Top/Shirt: Light Blue
  
• Bottom/Pants: Dark Brown
  
• Background: Light Green
  

then the model should consistently follow these colors throughout the entire shot.
If I notice that the color changes in a particular frame, I should be able to simply update the color (for example, change the shirt to Light Blue), and the model should automatically apply that corrected color consistently to the corresponding object across all frames in that shot.
This would give users an easy way to maintain temporal consistency while still allowing quick manual corrections whenever needed, without having to fix each frame individually.

[didris]

Hi Dan,

Thanks for the quick update!

There is just one issue. In the Fix Colors tab, under Target Image (B&W), it is still not possible to add multiple frames at once. They have to be selected one by one, which makes the workflow much slower. I hope this can be improved, because selecting each frame individually becomes quite tedious when working with many images.

For most videos, CMNET2 propagates the colors very well. However, there are also videos where it doesn't perform as consistently. If the same object has two or three different colors across different reference frames, it often either blends those colors together or, even worse, fails to propagate them correctly. I've attached two examples in case you're interested.

One question: do you think a Manual Edit tab would be technically feasible with the current pipeline? I mean something where the user could paint a mask over a small area (for example, a country flag) and recolor only that region while leaving the rest of the image untouched. Or would that require major changes to the DiT pipeline?

Thanks again for all your work!

https://www.swisstransfer.com/d/f313218b-1aa4-4232-b065-329fc4a38138

[Dan64]

Hi didris, 

  I enabled Ctrl+Click for multiple image selection in Browse. Update the project as described in the README

  You can try it yourself, but asking qwen-ie to color objects that aren't in the pictures won't work.
  For example asking to colorize Top/Shirt in Light Blue when in the image is missing a person dressing a top/shirt will produce artifacts.

  The prompts are memorized in a list so you don't have to re-insert the same prompt, it is just enough select it from the combo-box.

  It is not feasible to add mask management, but you can ask to qwen-ie to recognize the object. 
  For example you can ask to qwen-ie: "color the flag with the colors of Swiss flag"

Dan

[didris]

Hi Dan,
It works perfectly now. Thank you very much for your efforts!