Deoldify Vapoursynth filter

[didris]

This was just an attempt on my part to colorize
If there is another better way, of course I'd better implement it
My goal is to implement the best possible option for this type of video

[zspeciman]

Dan, your colorization job of Young Frankenstein is outstanding.  Those other two pale in comparison.  The stubborn red flickering are either nonexistent or not distracting.  This is the most significant update you've made.
I'm curious if you tested the stability with movies that have hardcoded subtitles, especially in dark scenes.  If the sudden appearance and disappearance of those bright white letters have any impact.

[Dan64]

This was just an attempt on my part to colorize
If there is another better way, of course I'd better implement it
My goal is to implement the best possible option for this type of video

Ok given that in this case it is possible recover some colors from the original clip, it is possible to use the following strategy:

1) create a colored clip using DDColor only and Retinex, this provides a colorful clip but with a lot of color noise
2) create a second clip with the colors recovered from the original clip, this will provide a more stable even if less colored clip (in practice this clip will substitute DeOldify)
3) combine the 2 clip created previously to obtain a colored clip with stable colors

here the clip the combine the strategy described above

# Imports
import vapoursynth as vs
# getting Vapoursynth core
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'D:/Programs/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# loading plugins
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
# Import scripts
import color
import vsdeoldify as havc
import validate
clip = core.lsmas.LWLibavSource(source="Crvena_Zvezda_Fiorentina_1966.mkv", format="YUV420P8", stream_index=0, cache=0, fpsnum=25, prefer_hw=0)
clip = clip[20:]   # skip test color
clip = core.std.Crop(clip=clip, left=20, right=20, top=4, bottom=4)# cropping to 1240x952
# Resizing using bicubic spline
clip = core.resize.Spline36(clip, width=1280, height=960)
frame = clip.get_frame(0)
# setting color matrix to 709.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT709)
# setting color transfer (vs.TRANSFER_BT709), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT709)
# setting color primaries info (to vs.PRIMARIES_BT709), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT709)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED)
# making sure frame rate is set to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive
# adjusting color space from YUV420P8 to RGB24 for vsHAVC
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="709", range_in_s="limited", range_s="full")
# create first clip using HAVC
clip1 = havc.HAVC_main(clip=clip, Preset="fast", ColorModel="DDColor(Artistic)", ColorMap="red->brown", ColorTune="medium", BlackWhiteTune="light", BlackWhiteMode=0, BlackWhiteBlend=True, ColorFix='Retinex/Red', EnableDeepEx=False)
clip1 = havc.HAVC_tweak(clip1, hue=10.00, sat=1.20, cont=0.95, bright=0)
# create the second clip using ColorAdjust and Tweak
clip2 = havc.HAVC_ColorAdjust(clip, BlackWhiteTune="strong", BlackWhiteMode=5, BlackWhiteBlend="True", ReColor=False)
clip2 = havc.HAVC_tweak(clip2, hue = 20, sat = 3.5, bright = 0, cont = 1.0, gamma = 1.0)
clip2 = havc.HAVC_adjust_rgb(clip2, strength=0.5, factor=(0.8, 1.10,1))
clip2 = havc.HAVC_bw_tune(clip2)
# merge the clips created previously, its possible to change the method (2-5) and/or the weight (0.2-0.8) to change the final colors (as preferred) 
clip = havc.HAVC_merge(clip1, clip2, method=3, weight=0.6)
clip = havc.HAVC_tweak(clip, sat = 1.05, cont = 0.95, hue = 5)
# adjusting output color from: YUV444PS to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="709", dither_type="error_diffusion")
# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# output
clip.set_output()

Dan

[didris]

--- removed quote ---

Hi, Dan64
1) I encoded the source with ddcolor and retinex
2) Now I encode the source via custom code

but regarding the third step and combining the two clips I'm not very clear how it will be done via custom code - there are no two videos in the code, but again only the black and white one?

PS: ColorFix='Retinex/Red' - in the latest release this function of the code is still missing

[Selur]

@Dan64: are you applying retinex (only) after the coloring? have you tried to apply it before the coloring. Applying deflicker and retinex seem to be options I would first apply before the coloring to kind of 'level the luma' the source and may be afterward to lessen color flickering,..

@didris: please, do not to use 'full quotes' unless it really makes sense. The previous post is still there, so 'full quotes' usually only hinder the general reading and understanding flow.

[Dan64]

I attached the zip archive with the new RC6
Inside the archive there is a folder named "vsdeoldify", this folder need to be extracted over your Hybrid installation, for example: .\Hybrid\64bit\Vapoursynth\Lib\site-packages 
so that the folder  "vsdeoldify" contained in  "site-packages" will be override with the new RC6.

You don't have to write any script, just copy this one

# Imports
import vapoursynth as vs
# getting Vapoursynth core
import sys
import os
core = vs.core
# Import scripts
import vsdeoldify as havc
# step1: read the clip
clip = havc.HAVC_read_video(source="Crvena_Zvezda_Fiorentina_1966.mkv")
# skip the first 20 frames with test color
clip = clip[20:]   
# crop bars
clip = core.std.Crop(clip=clip, left=20, right=20, top=4, bottom=4)# cropping to 1240x952
# Resizing using bicubic spline to 1280x960
clip = core.resize.Spline36(clip, width=1280, height=960)
# step2: adding colors using HAVC to clip1
clip1 = havc.HAVC_main(clip=clip, Preset="fast", ColorModel="DDColor(Artistic)", ColorMap="red->brown", ColorTune="medium", BlackWhiteTune="light", BlackWhiteMode=0, BlackWhiteBlend=True, ColorFix='Retinex/Red', EnableDeepEx=False)
clip1 = havc.HAVC_tweak(clip1, hue=10.00, sat=1.20, cont=0.95, bright=0)
# step3: adjusting color using HAVC_ColorAdjust/Tweak to clip2
clip2 = havc.HAVC_ColorAdjust(clip, BlackWhiteTune="strong", BlackWhiteMode=5, BlackWhiteBlend="True", ReColor=False)
clip2 = havc.HAVC_tweak(clip2, hue = 20, sat = 3.5, bright = 0, cont = 1.0, gamma = 1.0)
clip2 = havc.HAVC_adjust_rgb(clip2, strength=0.5, factor=(0.8, 1.10,1))
clip2 = havc.HAVC_bw_tune(clip2)
# merge clip1 & clip2, its possible to change the method (2-5) and/or the weight (0.2-0.8) to change the final colors (as preferred) 
clip = havc.HAVC_merge(clip1, clip2, method=3, weight=0.6)
clip = havc.HAVC_tweak(clip, sat = 1.05, cont = 0.95, hue = 5)
# adjusting output color to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="709", dither_type="error_diffusion")
# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# output
clip.set_output()

In this script, the merge is performed by the following function

# merge clip1 & clip2, its possible to change the method (2-5) and/or the weight (0.2-0.8) to change the final colors (as preferred) 
clip = havc.HAVC_merge(clip1, clip2, method=3, weight=0.6)

as suggested in the comment, you can try to change the parameter method (allowed values are: 2, 3, 4, 5) and/or the weight to get the clip with the best colors for you.

If all goes well, you should obtain a clip like this one: https://gofile.io/d/iiWtVr

Dan

[didris]

yes, this works Dan64, thanks for the help, 
but this is not the ideal coloring for this type of video, although it is not known at all whether it is possible to be ideal for a football match - the idea is to get the same picture for the entire match, without changing the color of the football field and the players' teams, because that way the impression remains that it is artificially colored). I hope that in time an even better way will be found than this - I will continue with the tests. I may also try with reference frames and it will become even better.

[Dan64]

Ok, if you want more color stability, you can add the following call

clip = havc.HAVC_ColorAdjust(clip, BlackWhiteTune="none", ReColor=True, Strength=4, ScThreshold=0.15, ScNormalize=False, DeepExVivid=False, ScMinFreq=0, chroma_resize=True)

Put it just before the call 

# adjusting output color to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="709", dither_type="error_diffusion")

In this way the color of the football field will be more stable (see picture)



Using  HAVC_ColorAdjust will slow down the encoding speed.
It is possible to speed-up the encoding speed by decreasing the size of the frames.
To do that change the following line

# Resizing using bicubic spline to 1280x960
clip = core.resize.Spline36(clip, width=1280, height=960)

with this line

# Resizing using bicubic spline to 940x720
clip = core.resize.Spline36(clip, width=940, height=720)

The previous RC6 had a problem in loading the scene detection plugin. 
Please use the RC7 attached to this post.

Dan

[didris]

Thank you, Dan64
i try tomorrow

[didris]

Hi, Dan64
I think the result with the new version is excellent - it's clear that this software has unlimited possibilities
https://archive.org/details/crvena-zvezd...new-method

I also tried it on an old, worn-out Houdini film
https://archive.org/details/houdini-source
the result is quite good, except that it seems to emphasize the purple color much more than necessary - but overall it did a pretty good job and even corrected some of the bad scenes

thanks again for the excellent work