Using Stable Diffision models for Colorization

[Dan64]

Hi, Dan

First of all, congratulations on the excellent work. The results are very impressive.
 
How do you manage to avoid inconsistent coloring across different scenes when the same objects appear?
For example, in one video I extracted 4,000 frames. To achieve consistent colors for the same objects, I had to manually remove about 3,800 of them. In some cases, a lady's clothing was colored in five different colors across different frames. If I let the process run automatically, those color variations remained visible throughout the movie.

How do you handle this issue?

Hi didris,

  In the case it could be useful I added a GUI to DiTServerRPC see post: #1.425

Dan

[didris]

Hi, Dan
great job, I will test it and write the results.

[didris]

Hi didris,

  In the case it could be useful I added a GUI to DiTServerRPC see post: #1.425

Dan

Hi, Dan

The installation was not easy, but I managed to get it working in the end. I think the problem was related to

tkinter_embed

in

requirements.txt

, which was causing issues in my environment. After removing it, everything installed correctly.

Overall, the software works now, and I successfully encoded five videos with durations of 10–15 minutes. Speed is 5 sec/frame with RTX5090. During testing, I noticed three issues:

1. It seems to work reliably only with MP4 and MKV files. I tested a MOV file, and it only started processing after I renamed it to MP4. Even then, it encoded only about 2 minutes and 30 seconds out of a 14-minute video.
   
2. The final output videos have no audio track.
   
3. In many videos, the reference frames were not colorized consistently, especially in the second half of the footage. In one particular video, almost none of the reference frames were colorized. I am not sure what this depends on or what type of source material works best. Perhaps a different Qwen model would help.
   
   

Despite these issues, this is a wonderful project and has a lot of potential. A few fixes would make it close to perfect.
I would also like to mention that with the first version I never had problems with frame colorization itself, apart from occasional inconsistencies where the same objects received different colors.

Congratulations on the great work, and thank you for making this available.

[safshe]

Getting below error in GUI

--- STARTING BATCH ---
⚠️ Error getting video info: CompletedProcess(args=['F:/AI_Works/DiTServerRPC/.venv/Lib/site-packages/vapoursynth/vspipe.exe', '--info', 'F:/AI_Works/DiTServerRPC/GUI/scripts\\vs_info.vpy', '-a', 'VideoPath=F:/AI_Works/DiTServerRPC/GUI/samples\\sample_bw.mp4'], returncode=1, stdout='', stderr="Failed to initialize VSScript. VSScript reported error: Python executable and library path couldn't be determined despite automatic configuration. Run `vapoursynth config` to set it for this Python installation and then try again.\n")
>>> STARTING TASK: EXTRACT
ℹ️ Starting extraction with script: "F:/AI_Works/DiTServerRPC/GUI/scripts\extract_refs_edge.vpy"
[EXTRACT] Failed to initialize VSScript. VSScript reported error: Python executable and library path couldn't be determined despite automatic configuration. Run `vapoursynth config` to set it for this Python installation and then try again.
⚠️ Error getting video info: CompletedProcess(args=['F:/AI_Works/DiTServerRPC/.venv/Lib/site-packages/vapoursynth/vspipe.exe', '--info', 'F:/AI_Works/DiTServerRPC/GUI/scripts\\vs_info.vpy', '-a', 'VideoPath=F:/AI_Works/DiTServerRPC/GUI/samples\\sample_bw.mp4'], returncode=1, stdout='', stderr="Failed to initialize VSScript. VSScript reported error: Python executable and library path couldn't be determined despite automatic configuration. Run `vapoursynth config` to set it for this Python installation and then try again.\n")
>>> STARTING TASK: COLORIZE
ℹ️ No images to colorize.
>>> STARTING TASK: ENCODE
⚠️ Error getting video info: CompletedProcess(args=['F:/AI_Works/DiTServerRPC/.venv/Lib/site-packages/vapoursynth/vspipe.exe', '--info', 'F:/AI_Works/DiTServerRPC/GUI/scripts\\vs_info.vpy', '-a', 'VideoPath=F:/AI_Works/DiTServerRPC/GUI/samples\\sample_bw.mp4'], returncode=1, stdout='', stderr="Failed to initialize VSScript. VSScript reported error: Python executable and library path couldn't be determined despite automatic configuration. Run `vapoursynth config` to set it for this Python installation and then try again.\n")
ℹ️ Starting encoding with script: "encode_cmnet2.vpy"
----------------------------------------------------------------
[ENCODE] "F:/AI_Works/DiTServerRPC/.venv/Lib/site-packages/vapoursynth/vspipe.exe" "F:/AI_Works/DiTServerRPC/GUI/scripts\encode_cmnet2.vpy" - -a "VideoPath=F:/AI_Works/DiTServerRPC/GUI/samples\sample_bw.mp4" -a "RefDir=F:/AI_Works/DiTServerRPC/GUI/samples\ref_qwen" -a "RenderSpeed=auto" -a "MemoryFrames=20" --outputindex 0 -c y4m | "F:/AI_Works/DiTServerRPC/GUI/tools/x265/x265.exe" --preset fast --input - --fps 24000/1001 --output-depth 10 --y4m --profile main10 --crf 20.00 --output "F:/AI_Works/DiTServerRPC/GUI/samples\sample_bw_cmnet2_dt-color.h265"
----------------------------------------------------------------
[COMPLETED] Encoding: F:/AI_Works/DiTServerRPC/GUI/samples\sample_bw.mp4 @ 24000/1001 fps
[COMPLETED] All tasks completed successfully.

[COMPLETED] All tasks completed.

[Selur]

Did you "Run `vapoursynth config` to set it for this Python installation and then try again." like the message suggested?

[safshe]

I tried to set it but it shows "no configuration needed"

[safshe]

Did you "Run `vapoursynth config` to set it for this Python installation and then try again." like the message suggested?

F:\AI_Works\DiTServerRPC>.\.venv\Scripts\activate

(.venv) F:\AI_Works\DiTServerRPC>vapoursynth config
No configuration needed!

(.venv) F:\AI_Works\DiTServerRPC>

[Dan64]

Try to set the windows user environment variable VSSCRIPT_PATH to

F:\AI_Works\DiTServerRPC\.venv\Lib\site-packages\vapoursynth\vsscript.dll

What script are you using to run the GUI ?
 
Dan

---

Hi, Dan

The installation was not easy, but I managed to get it working in the end. I think the problem was related to

tkinter_embed

in

requirements.txt

, which was causing issues in my environment. After removing it, everything installed correctly.

Overall, the software works now, and I successfully encoded five videos with durations of 10–15 minutes. Speed is 5 sec/frame with RTX5090. During testing, I noticed three issues:

1. It seems to work reliably only with MP4 and MKV files. I tested a MOV file, and it only started processing after I renamed it to MP4. Even then, it encoded only about 2 minutes and 30 seconds out of a 14-minute video.
   
2. The final output videos have no audio track.
   
3. In many videos, the reference frames were not colorized consistently, especially in the second half of the footage. In one particular video, almost none of the reference frames were colorized. I am not sure what this depends on or what type of source material works best. Perhaps a different Qwen model would help.
   
   

First all the GUI is very simple and is not a Hybrid replacement, it support only the colorization task, so if you need audio you need to mux it with mkvmerge.
To open the video stream I'm using LSmashSource I don't know the kind of support of MOV file, but if the problem is only this you can convert the MOV to MKV with mkvmerge.

Regarding the missing colorization, this is very strange.
Did you check the folder "ref_qwen" ? this folder contains the colored images, while the B&W images are stored in "ref_tht10", please check them.

Dan

[Selur]

LSMashSource => better use bs.VideoSource or lsmas.LWLibavSource

[safshe]

Try to set the windows user environment variable VSSCRIPT_PATH to

F:\AI_Works\DiTServerRPC\.venv\Lib\site-packages\vapoursynth\vsscript.dll

What script are you using to run the GUI ?
 
Dan

---

I did the above  VSSCRIPT_PATH solution but still same error.

I run gui by 

F:\AI_Works\DiTServerRPC>.\.venv\Scripts\activate

(.venv) F:\AI_Works\DiTServerRPC>python GUI\CMNET2_colorize_client_GUI.py