Keyframe Color Consistency Challenge Video Exemplar-Based Colorization Methods

[NASS]

Hello everyone,
I would like to open a discussion about an important challenge for exemplar-based colorization methods:
Keyframe Color Consistency Challenge
We need to find a reliable way to achieve temporal consistency between colorized keyframes so they can be used for the full colorization of black and white videos.
At the moment, one of the main limitations is that even when keyframes are individually colorized well, colors often vary from one keyframe to another. This creates flickering, unstable tones, skin tone changes, clothing color drift, and inconsistencies in objects or environments when processing the full sequence.
Why This Matters
For real-world video restoration, getting beautiful frame-by-frame results is not enough. We need:

• Stable skin tones from scene to scene
  
• Consistent clothing and object colors
  
• Lighting adaptation without color drift
  
• Smooth transitions between distant keyframes
  
• Reliable propagation across long sequences
  

Some Approaches I Tested
I experimented with several workflows using Automatic1111 + ControlNet + IP-Adapter.
Example Setup in Automatic1111
Main txt2img Prompt
Positive Prompt:

masterpiece, best quality, colorized photograph, vibrant, highly detailed

Negative Prompt:

black and white, grayscale, monochrome, bad quality, deformed

ControlNet Unit 0 (Preserve B&W Structure)

• Enable: Yes
  
• Pixel Perfect: Yes
  
• Input Image: Original black and white frame
  
• Control Type: Recolor
  
• Preprocessor:
  
  recolor_luminance
  or
  
  recolor_intensity
  
  
• Model:
  
  ioclab_sd15_recolor
  
  
• Weight:
  
  1.0
  (or
  
  0.8
  if too strong)
  

ControlNet Unit 1 (Inject Keyframe Colors)

• Enable: Yes
  
• Pixel Perfect: Yes
  
• Input Image: Previously colorized reference keyframe
  
• Control Type: IP-Adapter
  
• Preprocessor:
  
  ip-adapter_clip_sd15
  
  
• Model:
  
  ip-adapter_sd15
  
  
• Weight:
  
  0.8 – 1.0
  
  

This helps transfer colors from one keyframe to another while respecting the luminance of the target black and white frame.
But Problems Remain
Even with this method:

• Colors still drift over time
  
• Different scenes generate different palettes
  
• The same person may change skin tone
  
• Sky, grass, and buildings vary between shots
  
• Propagation becomes unstable in long videos
  

2 - Qwen-Image-Edit
From my tests, Qwen-Image-Edit currently delivers some of the best single-keyframe colorization results in terms of realism, detail, and natural palette generation.
However, there is still no practical way to maintain the same temporal consistency across all other keyframes in the video.
So even if one keyframe looks excellent, the next keyframe may still shift in:

• Skin tone
  
• Clothing colors
  
• Environment palette
  
• Object consistency
  
• Scene mood
  

Main Question
Could it be possible to use Qwen-Image-Edit for colorize all keyframes?
For example:

• Use one master keyframe style and propagate it
  
• Multi-keyframe palette memory
  
• Identity/object color locking
  
• Reference-aware batch processing
  
• Latent temporal consistency between generated keyframes
  

What Do You Think?
Is there a way to use Qwen-Image-Edit to colorize all keyframes and create one unified temporal color source for all reference images (keyframes)?
This could be a major breakthrough for AI video colorization.
Your turn — ideas, experiments, thoughts?

[Dan64]

Hi NASS,

   there is already a post in this forum regarding this topic opened by me: Using Stable Diffision models for Colorization

   I confirm that currently Qwen-Image-Edit is the best model for colorize B&W images. 
   I tried all the models available, including Fire-Image-Edit, but  Qwen-Image-Edit is much, much better.

   Currently my pipeline to colorize movies is the following:

    1) extract the reference frames (with HAVC)
        

clip = havc.HAVC_read_video(source=VideoPath)
clip = havc.HAVC_extract_reference_frames(clip, sc_threshold=0.035, sc_tht_ssim=0.8, sc_tht_offset=2, sc_min_int=20, sc_min_freq=15, sc_tht_white=0.70, sc_tht_black=0.09, sc_framedir=RefDir, sc_algo=1, ref_ext="jpg", ref_jpg_quality=95, ref_override=False, sc_debug=False)
# adjusting output color from RGB24 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="709", range_in_s="full", range_s="limited")

    2) colorize the reference frames with a DiT model (currently I'm using Nunchaku Qwen Image Edit 2511) 

prompt = "Colorize this image, natural colors. Strictly preserve all shapes, edges and background details."

   
    3) propagate the color of reference frames to full clip (with HAVC)

clip = havc.HAVC_read_video(source=VideoPath)
# adding colors using HAVC
clip = havc.HAVC_cmnet2(clip, method=4, sc_framedir=RefDir, encode_mode=0, render_speed="auto", max_memory_frames=50, ref_mode=0, render_vivid=False)
# adjusting output color from RGB24 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="709", range_in_s="full", range_s="limited", dither_type="error_diffusion")

   The films colorized in this manner achieve a cinema-grade quality
   I hope this helps.

Dan

P.S.
The HAVC_cmnet2() function that I'm using, is the one available in the coming version of HAVC 5.8.0, the one currently available in HAVC 5.6.7 was a my previous attempt of introducing the sliding window memory in ColorMNet, but given the lack of permanent memory It was not working, the produced colors were faded.

[NASS]

Hi Dan,

Thank you very much for sharing your workflow and your valuable findings. It is extremely interesting to see how far the pipeline has evolved, especially with the combination of high-quality reference frame extraction, Qwen-Image-Edit for keyframe colorization, and the new CMNET2 propagation stage.

Your results sound very impressive, and the fact that you describe them as cinema-grade quality says a lot about the progress being made. It is also very helpful to hear your comparison with other available models and your confirmation regarding Qwen-Image-Edit.

The improvements you mention with permanent memory and sliding window memory in CMNET2 sound particularly promising for solving the long-standing issues of temporal consistency and faded colors. That is exactly the kind of advancement many users have been hoping for.

I really appreciate you taking the time to explain everything in detail. I’m looking forward to trying the coming version of HAVC 5.8.0.

Thank you again, and best of luck with the release and future developments.

NASS