Deoldify Vapoursynth filter

[Dan64]

You are right. In effect I installed the packages in Hybrid with non version number and let pip to select the most appropriate version.
The version numbers that I included were the versions installed by pip.

opencv-contrib-python==4.9.0.80

  is the last version available in pip with a binary available.

As you know these AI programs require a lot of packages, I had to figure out what were the missing packages in Hybrid.
Fortunately were only 3, with opencv-contrib-python being the most important.

Dan

[Selur]

old:

def ddeoldify_main(clip: vs.VideoNode, Preset: str = 'Fast', VideoTune: str = 'Stable', ColorFix: str = 'Violet/Red',
                  ColorTune: str = 'Light', ColorMap: str = 'None', degrain_strength: int = 0,
                  enable_fp16: bool = True) -> vs.VideoNode:

def ddeoldify_stabilizer(clip: vs.VideoNode, dark: bool = False, dark_p: list = [0.2, 0.8], smooth: bool = False, smooth_p: list = [0.3, 0.7, 0.9, 0.0, "none"],
    stab: bool = False, stab_p: list = [5, 'A', 1, 15, 0.2, 0.15], colormap: str = "none", render_factor: int = 24) -> vs.VideoN

def ddeoldify(clip: vs.VideoNode, method: int = 2, mweight: float = 0.4, deoldify_p: list = [0, 24, 1.0, 0.0],
              ddcolor_p: list = [1, 24, 1.0, 0.0, True],
              dotweak: bool = False, dotweak_p: list = [0.0, 1.0, 1.0, False, 0.2, 0.5, 1.5, 0.5],
              ddtweak: bool = False, ddtweak_p: list = [0.0, 1.0, 2.5, True, 0.3, 0.6, 1.5, 0.5, "300:360|0.8,0.1"],
              degrain_strength: int = 0, cmc_tresh: float = 0.2, lmm_p: list = [0.2, 0.8, 1.0],
              alm_p: list = [0.8, 1.0, 0.15], cmb_sw: bool = False, device_index: int = 0,
              torch_dir: str = model_dir) -> vs.VideoNode:

new:

def HAVC_main(clip: vs.VideoNode, Preset: str = 'Fast', VideoTune: str = 'Stable', ColorFix: str = 'Violet/Red',
    ColorTune: str = 'Light', ColorMap: str = 'None', EnableDeepEx: bool = False, DeepExMethod: int = 0,
    DeepExPreset: str = 'Fast', ScFrameDir: str = None, ScThreshold: float = 0.1, ScMinFreq: int = 0,
    enable_fp16: bool = True) -> vs.VideoNode:

def HAVC_deepex(clip: vs.VideoNode = None, clip_ref: vs.VideoNode = None, method: int = 0, render_speed: char = 'medium', render_vivid: bool = True,
    enable_resize : bool = False, sc_framedir: str = None, only_ref_frames : bool = False, dark: bool = False, dark_p: list = [0.2, 0.8], smooth: bool = False, smooth_p: list = [0.3, 0.7, 0.9, 0.0, "none"],
    colormap: str = "none") -> vs.VideoNode:

def HAVC_ddeoldify(
        clip: vs.VideoNode, method: int = 2, mweight: float = 0.4, deoldify_p: list = [0, 24, 1.0, 0.0],
        ddcolor_p: list = [1, 24, 1.0, 0.0, True], ddtweak: bool = False,
        ddtweak_p: list = [0.0, 1.0, 2.5, True, 0.3, 0.6, 1.5, 0.5, "300:360|0.8,0.1"],
        cmc_tresh: float = 0.2, lmm_p: list = [0.2, 0.8, 1.0], alm_p: list = [0.8, 1.0, 0.15], cmb_sw: bool = False,
        sc_threshold: float = 0.0, sc_min_freq: int =0, device_index: int = 0, torch_dir: str = model_dir) -> vs.VideoNode:

def HAVC_stabilizer(clip: vs.VideoNode, dark: bool = False, dark_p: list = [0.2, 0.8], smooth: bool = False,
                        smooth_p: list = [0.3, 0.7, 0.9, 0.0, "none"], stab: bool = False,
                        stab_p: list = [5, 'A', 1, 15, 0.2, 0.15], colormap: str = "none",
                        render_factor: int = 24) -> vs.VideoNode:

=>
What happened to 'degrain_strength' ?
Why does HAVC_ddeoldify have sc_threshold and sc_min_freq but not EnableDeepEx and DeepExMethod?
So DeepEx  can only be used when using Presets ? (no custom ? Should Hybrid drop 'custom' ?)

Also: I don't see how to properly support DeepExMethod != 0 in Hybrid.

[Dan64]

I released a new version RC2 with a small update (see attachment).

I tried to be compatible as much as possible with the previous version 3.5.x

The main idea is to use HAVC_deolddify as a source of "exemplar" frames for "Deep-Exemplar-based Video Colorization" but at the same time to keep the use of HAVC_deolddify  in "standalone" mode as in the version 3.5.x. Also HAVC_deepex can using HAVC_deolddify as source or can work in "standalone" using only external "exemplar" frames. It is implemented (in RC2) also a mixed mode, where  HAVC_deepex  can use as sources both  HAVC_deolddify and external "exemplar" frames. 

I not only renamed the old filter functions (even if the old one are still available but are "deprecated") but for some of them I added/removed some parameters.

With the only exception of HAVC_stabilizer, which is the same as in version 3.5.x and need to be the last function in the rendering pipeline, I applied the following changes:


HAVC_ddeoldify

removed: dotweak, dotweak_p, degrain_strength (KNLMeansCL.dll not more necessary to be loaded)
added: sc_threshold, sc_min_freq (is necessary to load MiscFilters.dll)

if sc_threshold==0 no reference frames are generated and  HAVC_ddeoldify  will work in "standalone" mode as in version 3.5.x, so is not necessary to add a flag EnableDeepEx

HAVC_main

removed: degrain_strength 
added: EnableDeepEx, DeepExMethod, DeepExPreset, ScFrameDir, ScThreshold, ScMinFreq

The "custom" Preset need to be still supported, with the following logic:

If EnableDeepEx == False

The behavior is similar to the one already implemented in 3.5.x

If EnableDeepEx == True

There are 2 cases (introduced in RC2)

If DeepExMethod in (0,1,2)

   in the "custom" section need to be enabled the parameters of HAVC_ddeoldify and HAVC_deepex

   in this case the rendering pipeline will be:

    

clip_ref = HAVC_ddeoldify(clip, method=2, mweight=ddcolor_weight[w_id],
            deoldify_p=[0, deoldify_rf, 1.0, 0.0], ddcolor_p=[1, ddcolor_rf, 1.0, 0.0, enable_fp16],
            ddtweak=True, ddtweak_p=[0.0, 1.0, 2.5, True, 0.3, 0.6, 1.5, 0.5, hue_range],
            sc_threshold=ScThreshold, sc_min_freq=ScMinFreq)

clip_colored = HAVC_deepex(clip=clip, clip_ref=clip_ref, method=DeepExMethod, render_speed=DeepExPreset, render_vivid=True,
              enable_resize=False, sc_framedir=None, only_ref_frames=False, dark=True, dark_p=[0.2, 0.8],
              smooth=True, smooth_p=[0.3, 0.7, 0.9, 0.0, "none"], colormap=chroma_adjust)

If DeepExMethod in (3,4)
  
  HAVC_deepex will work in "standalone" using only the external reference frames (clip_ref = None) and in this case in the custom section need to be enabled only the parameters for HAVC_deepex

clip_colored = HAVC_deepex(clip=clip, clip_ref=None, method=DeepExMethod, render_speed=DeepExPreset, render_vivid=True,
                enable_resize=False, sc_framedir=ScFrameDir, only_ref_frames=False, dark=True, dark_p=[0.2, 0.8],
                smooth=True, smooth_p=[0.3, 0.7, 0.9, 0.0, "none"], colormap=chroma_adjust)

see the code of HAVC_main for a full example.

There is an interesting option implemented in HAVC_deepex that is "only_ref_frames" (available only in "custom" mode). If this option is enabled, in the folder specified by the parameter "sc_framedir" will be saved the reference frames used by  HAVC_deepex , since this option is added mainly for debugging in this case the clip is not colored by HAVC_deepex  and it will be still in black and white and only the reference frames will be colored. This option is useful not only for debug but also to manually adjust the bad colored frames. In this case it is possible to put in another directory only the bad frames with the fixed colors and eventually new reference frames and use the DeepExMethod=1 and set "sc_framedir" with the name of new folder containing the adjusted frames, to regenerate the video containing the "improved" frames. This feature was not available in version 3.5.x and thanks to "Deep-Exemplar-based Video Colorization" it is also possible to perform some kind of semi-automatic colorization.     

Dan

[Selur]

Okay,...
I do not seem get the whole 'DeepExMethod' thing.

:param DeepExMethod:        Method to use to generate reference frames.
                                        0 = HAVC (default)
                                        1 = HAVC + RF same as video
                                        2 = HAVC + RF different from video
                                        3 = external RF same as video
                                        4 = external RF different from video

RF = reference frame?

The main idea is to use HAVC_deolddify as a source of "exemplar" frames for "Deep-Exemplar-based Video Colorization" but at the same time to keep the use of HAVC_deolddify in "standalone" mode as in the version 3.5.x.

"exemplar" frames <= are these what you call RF ?

From the looks of it, I won't add 'DeepExMethod' and 'ScFrameDir' to Hybrid.
Don't see how to properly add these to Hybrid in a meaningful way while making sure they are used as intended.

=> I'll add "EnableDeepEx, DeepExPreset, ScThreshold, ScMinFreq" to HAVC_main.
These setting will not influence the 'custom'-mode at all.

I'll remove: dotweak, dotweak_p, degrain_strength from 'custom'-mode.

Cu Selur

[Dan64]

RF is the acronym of "reference frames" and yes it is equivalent to "exemplar frames".
I used this term because it is the term used in "Deep-Exemplar-based Video Colorization" 

You should rename the checkbox "DeOldify" at the left of "Coloring" panel in "HAVC" for the reasons already explained (the frame with the DeOldify parameter don't have to be renamed, it's Ok).

I suggest to move the "GPU ID" below Merge weight, so you have more space for DeepEx.

Having more space for DeepEx, you can create a dedicated "DeepEx" frame like "DeOldify" and "DDColor"

In this new frame you can add all the DeepEx parameters, including: method, render_speed, render_vivid, enable_resize, sc_framedir, only_ref_frames, dark, dark_p, smooth, smooth_p, colormap.

As you can note I added in DeepEx some of stabilization parameters, the reason is that when are used inside DeepEx these filters will be applied only on the reference frames and so are much faster.

If it is a problem the proper management of sc_framedir you can limit its usage by forcing the following convention:

if only_ref_frames is true
  sc_framedir = clip dir + "\RF_out"
if only_ref_frames is false and method > 0
  sc_framedir = clip dir + "\RF_in"

Once you have created this dedicated settings frame for "DeepEx" you can add also 'DeepExMethod', 'ScFrameDir' in the Preset section.

I know that this will complicate the GUI, take all the time you need, there's no rush.

Thanks,
Dan

[Selur]

Atm. one can either use:
1. a preset with HAVC_main which wraps: HAVC_ddeoldify + HAVC_deepex (optional) + HAVC_stabilizer
or
2. (custom) HAVC_ddeoldify + HAVC_stabilizer

Yes, I could adjust Hybrid to convert 'custom' to HAVC_ddeoldify + HAVC_deepex (optional) + HAVC_stabilizer, but I think this is overkill.
Like I mentioned time and time again, the is all getting too complex, too many parameters,...
Especially since I have to expect there to come an additional xy new parameters with the next release of HAVC/deloldify.

Since this is really bothering me, I thought about removing all the custom stuff and just limit Hybrid to support just the HAVC preset based part.
Problem is, that also annoys the hell out of me (only partially supporting havc/deoldify), I now decided to drop the support for it as a whole.

=> Next Hybrid version will not support HAVC/deoldify any more.
If folks want to use it, they would need to either write their own custom additions to Hybrid or write their own code in a custom section.

Cu Selur

[Dan64]

I really don't understand your answer.

I don't have intention to add any more parameters.

I can understand that modify the GUI to manage new parameters can be annoying.

But I don't see the reason to throw the baby out with the bathwater.

You have already developed the working code and the filter is working, probably is the most stable coloring filter available on public domain.

You can keep the current version with the limited support to DeepEx, and if one want to use more more complex settings can use the custom section.

Dan

[zspeciman]

Please continue the support for this filter. The amount of thought and creativity that went into this project and the results are astonishing.
There are many of us who are always excited to see the next upgrade to this wonderful tool.

[Selur]

1. I do not have the time and motivation to check the whole code of everything in __init__.py to figure out what inner workings and documentation changed.
2. The amount of parameters is too much, yet in each change it increases.
=> Sorry, it does not seem to be worth the effort for 2 users (Dan64, zspeciman).

[Dan64]

=> Sorry, it does not seem to be worth the effort for 2 users (Dan64,  zspeciman).

This post has about 13.532 visualizations so it seems that there are more people interested in this filter.