Deoldify Vapoursynth filter

[Selur]

Took a quick look,...
Since even the defaults changed and it has additional rules, it seems like a bad idea to add it to HAVC in Hybrid.
I'll probably leave HAVC as it is (aside from removing the RGB conversion) and add 'CMNet2Dit' as a separate filter listed under DDColor.

[Selur]

Uploaded a new Hybrid_havc_test for testing,...
(seems to work, but the filter isn't properly highlighted in the Filter Order/Queue, will look at that tomorrow)

[Dan64]

I do not get what "encode_mode = DeepExEncMode # default = 0 -> DeepExModel=0 (CMNET2) not exposed" is for.
DeepExEncMode can be 'remote' or 'local', isn't this given through the IP?

To colorize the reference frames is used the DiT Server, but to propagate the colors to all frames is used CMNET2, and CMNET2 can run both remote (encode_mode=0) or local ( encode_mode=1). The parameter IP is related to DiT Server, the CMNET server will run always locally and on random port.

Dan

[Selur]

Ah, okay.
Updated Hybrid_havc_test (fix: local vs remote in wrong order; added tool-tips)

Cu Selur

[Dan64]

An interesting feature that I added in HAVC_cmnet2dit() is the retry_threshold, if retry_threshold > 0, this value will be used to detect frames with insufficient permanent-memory coverage and in this case the frame with missing good reference will be colorized on the fly. The colorization model used is specified by retry_model. The best model to use in this case is 1 or 2 (depending on GPU used) but unfortunately this will slow down the inference speed by about 40%-50%. I added also the retry_model=0, in this case the frame will be colorized with HAVC (60% Deoldify + 40% DDColor), this is faster, but the quality is not good if compared to DiT models.

To be able to use retry_model=0 to colorize on the fly a single image I had to modify DDColor to add the class DDColorEngine which allow single image colorization.

I attached the new vsddcolor (zipped) wheel to this post, which contains this new functionality.

Dan

[Dan64]

Ah, okay.
Updated Hybrid_havc_test (fix: local vs remote in wrong order; added tool-tips)

Cu Selur

Wow!, a few months ago, it seemed impossible to add Diffusion/Transformes colorization to HAVC and today we are here testing it.


I tested your release and is good. Nice that in the GUI is now displayed the CPU and GPU usage.

I found the following small changes to be applied to the test version:

1) subsitute in the "CMNetDiT" Tool-tip

HAVCditEngine — a DiT-based model (Nunchaku/FLUX quantized)

with

HAVCditEngine — a DiT-based model (Nunchaku/Qwen-IE quantized)

I removed FLUX support due to the bad quality (I updated also the README).


2) add 3 decimal places to "SC tresh", given this treshold is applied to the edges is very sensitive to small shifts.


3) maybe is better to move the check-box "Vivid", below the spin-box "Frames"

4) in "Frames", change the Tool-tip in: "Default: 20, Min: 1 Max: 500"
  Also implement this range in the spin-box

5) in "SC min int", change the range in Tool-tip: "Range [1, 100]".
  Also implement this range in the spin-box

6) the "SC offset" should be applied to "SC min freq", the Tool-tip to be applied for this field is:
  "Offset index used for the Scene change detection. The comparison will be performed,
  between frame[n] and frame[n-offset]. An offset > 1 is useful to detect blended scene
  change, range[1, 25]. Default = 2".

 
I noticed that if I set max_memory_frames=20, everything works, but if I set max_memory_frames=50, it crashes, probably due to some timeout.
I couldn't get past 40 memory_frames.
In effect setting memory_frames=40 doesn't return any frames for at least 160/200 seconds.
I'll try to review the initial loading implementation.

Using max_memory_frames=20 I obtained an encoding speed of 8 fps (on a clip with 2543 frames), while using max_memory_frames=40 I obtained an encoding speed of 11 fps.

Thanks,
Dan

[Selur]

I noticed that if I set max_memory_frames=20, everything works, but if I set max_memory_frames=50, it crashes, probably due to some timeout.

Ah, I would have expected simply running out of vram would be the cause.

In effect setting memory_frames=40 doesn't return any frames for at least 160/200 seconds.

I guess you mean 160-200 (160 to 200) seconds. (not 160/200 = 0.8 seconds)

Adjusted the tool-tips and ranges.
=> updated Hybrid_havc_test


Cu Selur

[Dan64]

Fixed the problem regarding memory size. I attached the new RC3.
Now the initial preload is limited to 40 reference frames, the remaining reference frames will be loaded during the encoding.

I colorized a small clip with 2544 frames at a speed of 12.6 fps using max_memory_frames=50.

Your version is Okay, but are still missing the following changes:

1) in "SC min int", change the range in tool-tip: "Range [1, 100]".

2) the current tool-tip of "SC offset" should be applied to "SC min freq", the tool-tip to be applied to "SC offset" is:
           "Offset index used for the Scene change detection. The comparison will be performed,
             between frame[n] and frame[n-offset]. An offset > 1 is useful to detect blended scene
             change, range[1, 25]. Default = 2". 

Thanks,
Dan

[Selur]

updated Hybrid_havc_test, which should update the tool-tips.

Cu Selur

[NASS]

Hi Dan, I hope you're doing well, my friend. 

I'm still waiting for your response regarding the ColorAdjust ReColor Motion Distortion Issue. In the latest development version of cmnet, when I use the ReColor option to apply colorization to a clip that I've colorized externally, there's still this temporal consistency issue—it's like the image is jumping colors! Plus, the biggest problem is the distortion! I can fix the distortion issue with `chroma_resize=True`, but the result is still not as good as your original settings! 

Please, I’ve been waiting for your reply for days! Thanks