Using Stable Diffision models for Colorization - Printable Version

Using Stable Diffision models for Colorization - Printable Version

+- Selur's Little Message Board (https://forum.selur.net)
+-- Forum: Talk, Talk, Talk (https://forum.selur.net/forum-5.html)
+--- Forum: Small Talk (https://forum.selur.net/forum-7.html)
+--- Thread: Using Stable Diffision models for Colorization (/thread-4287.html)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14

RE: Using Stable Diffision models for Colorization - safshe - 17.06.2026

(17.06.2026, 15:05)Dan64 Wrote: As a natural extension of the "Fix Image" tab, I've added the new "Fix Video" tab.

Suppose you've already colorized a clip with CMNET2 and DitServerRPC. There will be a folder called "ref_qwen" containing the colored frames.
Suppose you've viewed the clip and want to change the color of some frames. Let's say these frames range from #34 to #525. After fixing them with the "Fix Image" tab, you can recolor the video only in the identified frame range (in this case, from #34 to #525). As shown in the image below.

I hope this helps; in some cases, it was necessary for me (which is why I added the implementation).

Since for technical reasons, I still need to re-encode the entire clip, it's mandatory to use as input the clip already colorized and to have NVEnC installed as explained in the README.

Dan

Please add a new feature to the Fix Image pipeline.
Proposed Workflow:

Step 1: Load a pre-colorized image to act as the primary color base.
Step 2: Select a new reference image that you want to colorize.
Step 3: Perform the colorization on the reference image, but strictly guide the colorization based on the palette and tones of the already loaded colorized frame.

I have tested a similar workflow in ChatGPT, and the logic works well. I hope we can implement this

RE: Using Stable Diffision models for Colorization - Dan64 - 17.06.2026

Hi safshe,

you must try with ComfyUi.
If it works with ComfyUi there is some possibility that it could be added in the GUI of DiTServerRPC.

Dan

RE: Using Stable Diffision models for Colorization - hallomanbh - 18.06.2026

While the model successfully colorizes and outputs all images, maintaining consistency across frames remains a major challenge. In many cases, the colors in the final colorized frame differ from those in the reference frame. As a result, the same shot may appear with varying colors across consecutive frames, leading to noticeable temporal inconsistencies and flickering in the video.

RE: Using Stable Diffision models for Colorization - Dan64 - 18.06.2026

If you extract a reference frame every 30-40 frames, flickering is very unlikely and this also improves color consistency.

At the end of reference frame's colorization I usually spend some time in checking the colorized frames in folder "ref_qwen".
In most cases it is just enough to delete the frames with different colors. Only in the the case are missing adjacent frames with correct colors, I'm using "Fix Image" to fix the color (this represent about the 10% of total frames deleted/removed).

In any case in Hybrid there are a lot of good filters to remove the flickering you can apply them at the end of colorization process.

Dan

RE: Using Stable Diffision models for Colorization - Dan64 - 19.06.2026

(20.05.2026, 21:12)Selur Wrote: About the paths,... Due to the changes in R75+ current Hybrid does load plugins from "Vapoursynth/Lib/site-packages/vapoursynth/plugins2" is they are available through pip. Main gain from this is that, Vapoursynth will, if the a dll has multiple versions (like for example zsmooth, vszip, hysteresis, cranexpr), automatically load the best suited when loading the base .dll (i.e. zsmooth.dll).
(I install/update the plugins through pip and move them to the plugins2 folder, to avoid autoloading.)
=> I plan to remove dlls which are in the plugins2 folder from vsfilters in the (near) future. (atm. they are still there)
Not sure whether I will switch to autoloading dlls in the future, but atm. that is not planned, so plugins2 folder is likely to stay for now.

side note: the next problematic thing will happen when Vapoursynth drops (atm. you just get annoying warning messages) API3 support,...
(hopefully this will not happen as fast as new Vapoursynth versions come out nowadays,...)

Cu Selur

I noted that some of plugins used by HAVC (see post: #44) are available now as plugins via pip.
The missing ones are:

1) TCanny (built using APIv4)
6) Retinex (built using APIv3)
7) SCDetect (built using APIv3)
8) ReduceFlicker (built using APIv3)
9) LSMASHSource (built using APIv3)

Regarding SCDetect, I noted that is available this plugin: vapoursynth-scenechange
Do you know it ?, do you think that can be a good replacement for SCDetect ?

How these plugings, installed via pip, can be checked and loaded inside a script ? can you provide an example ?

Another final question: I noted that the only plugin to reading clip/movies installed via pip is vapoursynth-bestsource, do you know if there are other APIv4 plugins that can be used in substitution of bestsource (still too slow on big movies) ?

Dan

RE: Using Stable Diffision models for Colorization - Selur - 19.06.2026

Plugins I install through pip (python -m pip install -U -r plugins.txt):

vapoursynth
vapoursynth-adaptivegrain
vapoursynth-akarin
vapoursynth-awarp
vapoursynth-bestsource
vapoursynth-bilateralgpu
vapoursynth-bm3d
vapoursynth-bwdif
vapoursynth-cranexpr
vapoursynth-d2vsource
vapoursynth-dctfilter
vapoursynth-deblock
vapoursynth-dedot
vapoursynth-descale
vapoursynth-descratch
vapoursynth-dghdrtosdr 
vapoursynth-dotkill
vapoursynth-edgefixer
vapoursynth-edgemasks
vapoursynth-eedi3
vapoursynth-fillborders
vapoursynth-fmtconv
vapoursynth-hysteresis
vapoursynth-iscombed
vapoursynth-knlmeanscl
vapoursynth-mvtools
vapoursynth-nlm-cuda
vapoursynth-nlm-ispc
vapoursynth-resize2
vapoursynth-sangnom
vapoursynth-scenechange 
vapoursynth-sneedif
vapoursynth-subtext
vapoursynth-timecube
vapoursynth-vivtc
vapoursynth-vszip
vapoursynth-wnnm
vapoursynth-zit
vapoursynth-znedi3
vapoursynth-zsmooth
vs-placebo
vsnoise

and move them (from "Vapoursynth\Lib\site-packages\vapoursynth\plugins") into "Vapoursynth\Lib\site-packages\vapoursynth\plugins2" to avoid automatic loading.
here's the content of the plugins2 folder:

Vapoursynth\Lib\site-packages\vapoursynth\plugins2>ls -R
.:
DGHDRtoSDR.dll  bilateralgpu   deblock.dll    eedi3m.dll         libdescale.dll      sneedif         vszip
KNLMeansCL.dll  bm3d.dll       dedot.dll      fillborders.dll    libvs_placebo.dll   subtext.dll     wnnm.dll
adaptivegrain   bwdif.dll      descratch.dll  fmtconv.dll        mvtools.dll         timecube.dll    zit.dll
addnoise.dll    cranexpr       dotkill.dll    hysteresis         nnedi3_weights.bin  vivtc.dll       znedi3.dll
akarin          d2vsource.dll  edgefixer.dll  iscombed.dll       resize2.dll         vsnlm_cuda.dll  zsmooth
awarp.dll       dctfilter.dll  edgemasks.dll  libbestsource.dll  sangnom.dll         vsnlm_ispc.dll

'.\adaptivegrain':
adaptivegrain_rs.avx2.dll  adaptivegrain_rs.dll  adaptivegrain_rs.zn4.dll  manifest.vs

'.\akarin':
libakarin.dll  libzstd.dll

'.\bilateralgpu':
bilateralgpu.dll  bilateralgpu_rtc.dll

'.\cranexpr':
cranexpr.avx2.dll  cranexpr.dll  cranexpr.zn4.dll  manifest.vs

'.\hysteresis':
hysteresis.dll  hysteresis.v2.dll  hysteresis.v3.dll  manifest.vs

'.\sneedif':
libsneedif.dll  nnedi3_weights.bin

'.\vszip':
manifest.vs  vszip.avx2.dll  vszip.dll  vszip.zn4.dll

'.\zsmooth':
manifest.vs  zsmooth.avx2.dll  zsmooth.dll  zsmooth.zn4.dll

Quote:Do you know it ?, do you think that can be a good replacement for SCDetect ?

Works fine here. (is used by Hybrid, vsscripts are adjusted for it)
I always call misc.SCDetect, which uses a plugin if it's available and falls back to a manual scene change detection otherwise.

Quote:Another final question: I noted that the only plugin to reading clip/movies installed via pip is vapoursynth-bestsource, do you know if there are other APIv4 plugins that can be used in substitution of bestsource (still too slow on big movies) ?

DGDecNV depending on the source. Aside from BestSource all the other libav based source filters are still APIv3 since nobody cared enough to update them to APIv4.

Cu Selur

RE: Using Stable Diffision models for Colorization - Dan64 - 20.06.2026

(17.06.2026, 16:54)safshe Wrote: Please add a new feature to the Fix Image pipeline.
Proposed Workflow:
Step 1: Load a pre-colorized image to act as the primary color base.

Step 2: Select a new reference image that you want to colorize.

Step 3: Perform the colorization on the reference image, but strictly guide the colorization based on the palette and tones of the already loaded colorized frame.

I have tested a similar workflow in ChatGPT, and the logic works well. I hope we can implement this

I added the new Tab "Fix Colors" to the project HAVCServerDiT

[Image: attachment.php?aid=3660]

[Image: attachment.php?aid=3660]

Using this tab it is possible to propagate the colors from the Reference Image (on the left) to the Target Image (in the middle). The resulting colorized image is shown on the right.

The best model to perform this color propagation is CMNET2 which is the model used for this colorization. In case the color propagation is not satisfactory, it is possible to use the button "Copy -> Fix Image" and the colorized image is provided as input to the Tab "Fix Image" where it will be possible to use Qwen-Image-Edit to apply additional color adjustments.

Dan

P.S.
Please update the code as described in the README

RE: Using Stable Diffision models for Colorization - safshe - 22.06.2026

(20.06.2026, 23:38)Dan64 Wrote:
(17.06.2026, 16:54)safshe Wrote: Please add a new feature to the Fix Image pipeline.
Proposed Workflow:
Step 1: Load a pre-colorized image to act as the primary color base.

Step 2: Select a new reference image that you want to colorize.

Step 3: Perform the colorization on the reference image, but strictly guide the colorization based on the palette and tones of the already loaded colorized frame.

I have tested a similar workflow in ChatGPT, and the logic works well. I hope we can implement this

I added the new Tab "Fix Colors" to the project HAVCServerDiT

Using this tab it is possible to propagate the colors from the Reference Image (on the left) to the Target Image (in the middle). The resulting colorized image is shown on the right.

The best model to perform this color propagation is CMNET2 which is the model used for this colorization. In case the color propagation is not satisfactory, it is possible to use the button "Copy -> Fix Image" and the colorized image is provided as input to the Tab "Fix Image" where it will be possible to use Qwen-Image-Edit to apply additional color adjustments.

Dan

P.S.
Please update the code as described in the README

you tried this ? https://huggingface.co/DoctorDiffusion/LTX-2.3-IC-LoRA-Colorizer

RE: Using Stable Diffision models for Colorization - Dan64 - 22.06.2026

Hi safshe,

LTX-2.3-22b-IC-LoRA-Colorization is a LoRA for LTX-2.3, there is a more recent version here: Lightricks/LTX-2.3-22b-IC-LoRA-Colorization
LTX-2.3 is a Juggernaut, minimum HW requirements is 32GB+ VRAM, well beyond my 16GB VRAM.
I'm not planning to try it in the near future.

Dan

RE: Using Stable Diffision models for Colorization - didris - 26.06.2026

(22.06.2026, 19:12)Dan64 Wrote: Hi safshe,

LTX-2.3-22b-IC-LoRA-Colorization is a LoRA for LTX-2.3, there is a more recent version here: Lightricks/LTX-2.3-22b-IC-LoRA-Colorization
LTX-2.3 is a Juggernaut, minimum HW requirements is 32GB+ VRAM, well beyond my 16GB VRAM.
I'm not planning to try it in the near future.

Dan

Hi, Dan
maybe this help you for 16 gb vram -

Quote:https://github.com/deepbeepmeep/Wan2GP/tree/main