Deoldify Vapoursynth filter

[Dan64]

I introduced the "auto" to help the beginners to assign a render_factor.
But this parameter is available only for ddcolor, also the calculated render_factor could impact significantly the speed.

In my opinion it could be removed.

In case you decide to add it, the calculated value should be visible in the render_factor box (gray-out) so that the user can see what render_factor is used.

The algo is the following

ddcolor_rf = min(max(math.trunc(0.4 * clip.width / 16), 16), 48)

Dan

[Selur]

will look at it later

[Dan64]

I like the way the new GUI is managing the parameters.

Given that in this way the complexity of managing the parameters is decreased.
I decided to reintroduce the merging method: Adaptive Luma Merge

5 : Adaptive Luma Merge:
                        given that the ddcolor() perfomance is quite bad on dark scenes, the images are
                        combined by decreasing the weight assigned to ddcolor() when the luma is
                        below a given threshold given by: luma_threshold. The weight is calculated using
                        the formula: merge_weight = max(mweight * (luma/luma_threshold)^alpha, min_weight).
                        For example with: luma_threshold = 0.6 and alpha = 1, the weight assigned to
                        ddcolor() will start to decrease linearly when the luma < 60% till "min_weight".
                        For alpha=2, begins to decrease quadratically (because luma/luma_threshold < 1).   

With the following parameters

:param alm_p:   parameters for method: "Adaptive Luma Merge" (see method=5 for a full explanation)
                   [0] : luma_threshold: threshold for the gradient merge, range [0-1] (0.01=1%)
                   [1] : alpha: exponent parameter used for the weight calculation, range [>1]
                   [2] : min_weight: min merge weight, range [0-1] (0.01=1%)

Are only 3 parameters.

Just to reassure you, I have no intention to add in the future also the "chroma_limiter" since the new chroma stabilizer is working well there is no reason to add another temporal filter.

Just for curiosity I have an hardware device that I use to check the "chroma" stability of a movie.

It is a "Fire TV Cube 4K (2^ gen)".

I never had problems to play movies with this device, till I started to use DDColor to colorize the movies.
The are part of movies colored with DDColor that shows  a disturbing psychedelic flashing effect that is so strong that this device is unable to reproduce the colors and shows blocking square box in the areas of the movie where happen fast changes of colors.

Among all the filters developed to stabilize DDColor, only the filter "stab" with 5 frames is able to produce a movie colored with DDColor which plays smoothly on my  "Fire TV Cube 4K (2^ gen)". Human eyes are much less sensitive to imperfections in chrominance compared to luminance. And probably the stabilization effect of this filter is not visible with our eyes, but my device see them and allows me to also see them when the device starts showing rendering problems.        

Dan

[Selur]

I'm not really keen on adding additional options.

Does having four different stabilization methods make any sense?

:param method:              method used to combine deoldify() with ddcolor() (default = 2):
                                    0 : deoldify only (no merge)
                                    1 : ddcolor only (no merge)                                
                                    2 : Simple Merge (default):
                                        the frames are combined using a weighted merge, where the parameter "mweight"
                                        represent the weight assigned to the colors provided by the ddcolor() frames.                                    
                                    3 : Constrained Chroma Merge:
                                        given that the colors provided by deoldify() are more conservative and stable
                                        than the colors obtained with ddcolor(). The frames are combined by assigning
                                        a limit to the amount of difference in chroma values between deoldify() and
                                        ddcolor() this limit is defined by the threshold parameter "cmc_tresh".
                                        The limit is applied to the image converted to "YUV". For example when
                                        cmc_tresh=0.2, the chroma values "U","V" of ddcolor() frame will be constrained
                                        to have an absolute percentage difference respect to "U","V" provided by deoldify()
                                        not higher than 20%.  
                                    4 : Luma Masked Merge:  
                                        the frames are combined using a masked merge, the pixels of ddcolor() with luma < "luma_mask_limit"
                                        will be filled with the pixels of deoldify(). If "luma_white_limit" > "luma_mask_limit" the mask will
                                        apply a gradient till "luma_white_limit". If the parameter "mweight" > 0 the final masked frame will
                                        be merged again with the deoldify() frame.
                                    5 : Adaptive Luma Merge:
                                        given that the ddcolor() perfomance is quite bad on dark scenes, the images are
                                        combined by decreasing the weight assigned to ddcolor() when the luma is
                                        below a given threshold given by: luma_threshold. The weight is calculated using
                                        the formula: merge_weight = max(mweight * (luma/luma_threshold)^alpha, min_weight).
                                        For example with: luma_threshold = 0.6 and alpha = 1, the weight assigned to
                                        ddcolor() will start to decrease linearly when the luma < 60% till "min_weight".
                                        For alpha=2, begins to decrease quadratically (because luma/luma_threshold < 1).

shouldn't there be just one?

Cu Selur

Ps.: side note: can you remove the __pycache__-folders before zipping the vsdeoldify-xxx attachments?

[Selur]

ddcolor_rf = min(max(math.trunc(0.4 * clip.width / 16), 16), 48)

I realized I can not reflect this in the user interface easily, since I don't know the resolution of the video in the ui.
When you move the filter, or change cropping, resize, letterbox,... the input resolution to the filter can change.
I adjusted Hybrid so that when calling ddeoldify_stabilizer ddColorRF will be calculated properly if it was set to 0.
(updated the download)

CuSelur

[Selur]

:param alm_p:               parameters for method: "Adaptive Luma Merge" (see method=5 for a full explanation)
                                   [0] : luma_threshold: threshold for the gradient merge, range [0-1] (0.01=1%)
                                   [1] : alpha: exponent parameter used for the weight calculation, range [>1]
                                   [2] : min_weight: min merge weight, range [0-1] (0.01=1%)

+

alm_p: list = [0.3, 0.6, 1.0]

I susect that

[1] : alpha: exponent parameter used for the weight calculation, range [>1]

with ">1" should be ">0"

Cu Selur

Ps.: updated download

[Dan64]

I'm not really keen on adding additional options.

Does having four different stabilization methods make any sense?

shouldn't there be just one?

 
Each stabilization method approach the problem from a different point of view.

I discovered that: contrast, luma, gamma, if are changed before the colorize process, can affect significantly the output of colors.

Unfortunately is not possible provide a one size fits-all solution and the methods and filter parameters need to be adjusted depending on the type of video to be colored.
To colorize perfectly a movie it will be necessary split it in several clips so that for each clip it will be possible to apply the best settings.

The 4 methods, including  "Adaptive Luma Merge" should cover the most frequent situations.

So please add it.

Thanks,
Dan

[Selur]

test the latest download it should include the option

[Dan64]

[1] : alpha: exponent parameter used for the weight calculation, range [>1]

with ">1" should be ">0"

Cu Selur

Ps.: updated download

Actually should be range [>=1], because (by definition) luma/luma_threshold < 1 a base with exponent >1 will decrease the value. 
Since the scope is to decrease the weight linearly (alpha=1) or exponentially (alpha>1) alpha must be >= 1.

Dan

---

test the latest download it should include the option

In the dev-version of 31/03/24 12:23 the filter was missing.

Dan

---

ddcolor_rf = min(max(math.trunc(0.4 * clip.width / 16), 16), 48)

I realized I can not reflect this in the user interface easily, since I don't know the resolution of the video in the ui.
When you move the filter, or change cropping, resize, letterbox,... the input resolution to the filter can change.
I adjusted Hybrid so that when calling ddeoldify_stabilizer ddColorRF will be calculated properly if it was set to 0.
(updated the download)

CuSelur

I thought about it and it's better to remove "auto" this will simplify the code and will force the user to select a render_factor.
As explained previously I added it at beginning because I substitute "input_size" with "render_factor".
But thanks to the exhaustive read-me written in the filter release, the users should understand what is the render_factor and so "auto" is not more necessary.

For once I'm asking you to remove something instead of adding it.

Dan

[Selur]

In the dev-version of 31/03/24 12:23 the filter was missing.

uploaded the file to a different folder should work now.
I'll remove the 'auto' option later.

Actually should be range [>=1], because (by definition) luma/luma_threshold < 1 a base with exponent >1 will decrease the value.

okay, then the function definition:

clip: vs.VideoNode, method: int = 2, mweight: float = 0.4, deoldify_p: list = [0, 24, 1.0, 0.0], ddcolor_p: list = [1, 24, 1.0, 0.0, True], ddtweak: bool = False, ddtweak_p: list = [0.0, 1.0, 1.0, False, 0.4, 0.4],  cmc_tresh: float = 0.2, lmm_p: list = [0.3, 0.6, 1.0], alm_p: list = [0.3, 0.6, 1.0], dark: bool = False, dark_p: list = [0.2, 0.8], cmb_sw: bool = False, device_index: int = 0, torch_dir: str = model_dir) -> vs.VideoNode:

neeeds to be fixed. There (alm_p: list = [0.3, 0.6, 1.0]) "alpha" is 0.6,...