Deoldify Vapoursynth filter

[Dan64]

Hello Selur,
 
Please find attached the release candidate RC1 of new HAVC release 5.0: vsdeoldify-5.0.0_RC1.zip
In this release there are many improvements and some of them will have an impact also on the GUI. To help to understand better the changes on the code I provided some example in the attachment HAVC5_samples.zip

The main changes are the following:

----------------------------------------------------------------------
1) NEW EXEMPLAR-BASED MODEL : DeepRemaster
----------------------------------------------------------------------

First all, I found an interesting way to add DeepRemaster to HAVC. To use it is necessary to download the file remasternet.pth.tar (is not a tar, just a "pth" renamed as "pth.tar") and copy it in: ".\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\remaster\model". 

In the GUI the new model can be added in the “Exemplar Models” section as “ColorMnet” and “Deep-Exemplar”.

Like ColorMNet is possible to specify the max number of reference frames to keep in memory, but in this case the number is limited to 10-20 max 50. The problem is that more reference frames will be used in the inference more time will be necessary to estimate the colored image. Like BestSource I need to scan all the clip to get all the reference frames, because in my implementation of Deep-Remaster are provided in input for the inference 50% of past frames and 50% of future frames and to do that I need to scan the clip for the reference frames. 
To don’t slow down too much the start of the encoding I implemented a buffered scan where the frames are read in batch during the coloring process. 

This method can be used  to colorize a movie with a style that tries to maintain the visual impact of original black and white version and could be interesting for some kind of movie.

Here an example of movie colored using DeepRemaster: The Trial (colorized, 1962)

Here some sample code:

Impact on HAVC_deepex "ex_model":
  0 : ColorMNet (default)
  1 : Deep-Exemplar
  2 : Deep-Remaster

see examples:

HAVC_main_deepremaster_extRF.vpy : example of using DeepRemaster with only external reference
frames

HAVC_main_deepremaster_HAVC.vpy : example of using DeepRemaster with HAVC

HAVC_main_deepremaster_HAVC+extRF.vpy : example of using DeepRemaster with HAVC
and external reference frames

Impact on HAVC_main "DeepExModel":
  0 : ColorMNet (default)
  1 : Deep-Exemplar
  2 : Deep-Remaster

see example

HAVC_main_deepremaster_HAVC_merge.vpy : example of using DeepRemaster with HAVC
and ref_merge > 0

----------------------------------------------------------------------
2) COLORING MODEL: HAVC and DeOldify (Only)   
----------------------------------------------------------------------

With the addition of DeepRemaster now there are 7 models implemented in HAVC and I think that is time to rename the filter DeOldify in HAVC as shown in the following picture



 
and to add the original version of DeOldify like DDcolor as shown in the following picture



Here some sample code:

HAVC_deoldify_only.vpy : example to call DeOldify (original version)
since the previous DeOldify will be called HAVC it is right to provide
also the color model DeOldify (only) as DDcolor.

----------------------------------------------------------------------
3) NEW USE of Method=5 "HAVC different_from_video"
----------------------------------------------------------------------

When is select the methd "HAVC different_from_video" is necessary to
enable "Ref FrameDir" to select a video clip like in vs-propainter. This
method is useful to restore some old colored movies or movies colored with DeOldify

In this case the clipRef must use the external video selected using
"Ref FrameDir", and Hybrid need to provide the necessary conversion code to
render clipRef compatible with HAVC.

Here some sample code:

HAVC_deepremaster_method=5.vpy : example of using DeepRemaster with method 5


It has been added an additional parameter ref_freq in HAVC_deepex to manage better the method = 5
It need to be added in the region 3 of the big picture above.

:param ref_freq:            If (ref_merge > 0 or method == 5). If > 0 will be generated at least a reference frame
                                every "ref_freq" frames. range [0-1500]. If is not set, is assigned automatically a
                                value depending on ref_merge value and method.

----------------------------------------------------------------------
4) NEW ColorMNet encode_mode=2 -> "remote all-ref"
----------------------------------------------------------------------

In HAVC 5.0 has been introduced the possibility to provide to ColorMNet all
the reference frames at the beginning of coloring process. Its use is quite
transparent for Hybrid. The main usage is with the method=5.
This method need to be added in the GUI in the region 3 and will extended the encode_mode parameter

:param encode_mode:         Parameter used by ColorMNet to define the encode mode strategy.
                                Available values are:
                                     0: remote encoding. The frames will be colored by a thread outside Vapoursynth.
                                                         This option don't have any GPU memory limitation and will allow
                                                         to fully use the long term frame memory.
                                                         It is the faster encode method (default)
                                     1: local encoding.  The frames will be colored inside the Vapoursynth environment.
                                                         In this case the max_memory will be limited by the size of GPU
                                                         memory (max 15 frames for 24GB GPU).
                                                         Useful for coloring clips with a lot of smooth transitions,
                                                         since in this case is better to use a short frame memory or
                                                         the Deep-Exemplar model, which is faster.
                                     2: remote all-rf    Same as "remote encoding" but all the available reference frames
                                                         will be used for the inference at the beginning of encoding.

 
Here some sample code:

HAVC_colormnet_method=5.vpy : example of using method=5 with ColorMNet


----------------------------------------------------------------------
5) NEW PRESET: BlackWhiteTune
----------------------------------------------------------------------

This preset need to be added in the region 1 of the big picture above.

Impact on HAVC_main "BlackWhiteTune":

:param BlackWhiteTune:      This parameter allows to improve contrast and luminosity of Black & White input clip to
                                be colored with HAVC. Allowed values are:
                                    'None' (default)
                                    'Light',
                                    'Medium',
                                    'Strong'

 


Here some sample code: 

HAVC_deoldify_bw_tune_preset.vpy : example of using the new preset BlackWhiteTune

In the custom section is Hybrid that must generate the coide for usuing it.

HAVC_deoldify_bw_tune.vpy : example to apply the preset BlackWhiteTune in the custom section
in this case is necessary to cal the function

clip = HAVC_bw_tune(clip, "strong", "on")  -> at the start of coloring process

and

clip = HAVC_bw_tune(clip, "strong", "off") -> at the end of coloring process


I hope that you can incorporate the described changes in Hybrid.

Thanks,
Dan