Using Stable Diffision models for Colorization

[buckinpgh]

Hi Dan and Selur. Love everything you guys have done. Please tell me if this is the wrong place to ask or move as necessary. 

I have pretty static b&w video (think an interview where the camera doesn't move and subjects don't move much), so scene detection captures only a frame or 2 over a 10 minute video. I've tried extracting 1 frame out of every 30 with ffmpeg as well as making a slight modification to your DiT server GUI to allow me to bypass scene detection and extract 1 frame for every 30 (always preserving a good ref 0). Filenames are correct format. I'm then batch colorizing them using qwen.  I've compared the b&w image to the colorized image and they look like the frame numbers are correct (which is why I tried extracting both ways). The images are perfect after coming out of qwen or i re-do them individually until all are good. I put the images in the ref_qwen directory for the video to be colorized, but the video colorization does not seem to be respecting the reference frames. For instance, if the interviewee is wearing a black shirt in all reference frames the shirt color will fluctuate (usually a dark orange), background too, even though nothing ever changes in the refs. skin tones also change, but not in refs. Same results in HAVC through Hybrid being sure to feed it the reference folder. I'm obviously doing something wrong because if I feed the Casablanca HD trailer through DiT server it comes out great, including if I follow my same workflow of extracting through DiT or ffmpeg, colorizing through qwen, then encoding through DiT. Running my videos start to finish through DiT gives bad results which is why I assumed it was limited number of refs due to lack of 'scenes'. Here's the script DiT server is using for extraction (after modification to skip scene detection). I'm running a batch reference job with qwen right now but when done I'll feed it through DiT and can give the encoding script being used, just wanted to see if you had any suggestions or anything glaring that I'm missing in the meantime. Not that I think it matters but I'm running on an RTX 5060Ti 16GB.

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
import vapoursynth as vs
import vscmnet2 as cmnet2

core = vs.core

# Dummy paths for troubleshooting
VideoPath = r"C:\input\example_video.mp4"
RefDir = r"C:\output\ref_every30"

# Read the source video
clip = cmnet2.vs_read_video(source=VideoPath)

# Export frame 0, 30, 60, 90, ...
# In vscmnet2, ref_list=[30] means "export every 30 frames"
clip = cmnet2.vs_export_list_frames(
    clip,
    sc_framedir=RefDir,
    ref_list=[30],
    offset=0,
    ref_ext="jpg",
    ref_jpg_quality=95,
    ref_override=True,
    fast_extract=True,
)

# Pass-through output so vspipe can drive evaluation
clip = core.resize.Bicubic(
    clip=clip,
    format=vs.YUV420P10,
    matrix_s="709",
    range_in_s="full",
    range_s="limited",
)

clip.set_output()
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[Dan64]

Hi buckinpgh,

  your script looks good, but note that RefDir must be in the same folder of Video clip.
  In your case you must set it to

RefDir = r"C:\input\ref_qwen"

 but you don't have to override these values, so the correct filter should be written in this way

import vapoursynth as vs import vscmnet2 as cmnet2 core = vs.core # Read the source video clip = cmnet2.vs_read_video(source=VideoPath) # Export frame 0, 30, 60, 90, ... # In vscmnet2, ref_list=[30] means "export every 30 frames" clip = cmnet2.vs_export_list_frames(     clip,     sc_framedir=RefDir,     ref_list=[30],     offset=0,     ref_ext="jpg",     ref_jpg_quality=95) # Pass-through output so vspipe can drive evaluation clip = core.resize.Bicubic(     clip=clip,     format=vs.YUV420P10,     matrix_s="709",     range_in_s="full",     range_s="limited", ) clip.set_output()

 moreover you should rename the script as "extract_refs_every30.vpy" and put it in the folder scripts (please left unchanged the original script extract_refs_edge_ext.vpy).

 after this change you need to restart the GUI and now you should be able to select the new script in the combo-box "Extract VPY:"

 If the colors looks blended or faded, try to lower the memory frames to 10.

Dan
   

 

 moreover,

[buckinpgh]

Thanks Dan. The input and output folders were just dummy values, my ref folder is in the same folder as the video clip and named ref_qwen in the same folder after colorizing. I'm going to do a run with a large number of reference frames (like 1 every 5) and try doing the encode via script rather than the gui to have a bit more control over things. I've used colormnet with pre-masked objects/people a lot with very good results, but it's time consuming. I now have an excellent qwen batch pipeline where I can set it to run over hundreds of frames and only have to re-do a handful so if I can get the encoding right it will save a ton of time.

I get I'm probably a special use case. For cmnet's intended purpose of colorizing old movies where there's a lot of movement and scene changes it's excellent and I can run it start to finish in the gui and get amazing results. 

Thanks for all your work! I'm having a lot of fun just playing around with it. I'll repost after my next run with the results and encode script I used.