28.03.2024, 16:39 (This post was last modified: 28.03.2024, 16:39 by Selur.)
DDConrol, on my system with a 1000x542 source:
16bit (FP16 enabled) ~37.69fps
32bit (FP16 disabled) ~22.9fps
but speeds will differ depending on the used card and input.
(28.03.2024, 15:51)Selur Wrote: @Dan64: Maybe add an FP16 option for DDColor, depending on which DDColor is called with RGBS or RGBH content?
Cu Selur
I will add this option in the next release
Dan
(27.03.2024, 20:28)Selur Wrote: Sounds good.
Have to taken to handle scene changes properly?
Cu Selur
I added also a threshold to detect scene changes, if the average of pixels substituted is above this threshold (for example 80%) it will be considered a scene change and the restored -/+ frame will be replaced by the current frame (eventually de-saturated).
I completed the changes for the new version 3.0.0.
On filtering side I improved the "chroma stabilizer" to solve the problem related to gray areas in the past/future frames.
I also added the requested changes on ddcolor regrading the optional fp16 and delayed loading of the filter.
But the big changes are on the interface of the filter that has changed significantly.
As agreed I removed the post-process stabilization filters in another filter called: ddeoldify_stabilizer
The new header for ddeoldify is
def ddeoldify(
clip: vs.VideoNode, color_method: int = 2, merge_weight: float = 0.4, deoldify_params: list = [0, 24, 1.0, 0.0], ddcolor_params: list = [1, 24, 1.0, 0.0], ddcolor_tweaks: list = [False, 0.4, 0.4, 0.0, 1.0, 1.0, True, 1], combine_params: list = [False, 0.6, 2.0, 0.15, 0.2, 0.3, 0.6, 1.0], darkness: list = [False, 0.2, 0.8], chroma_resize: bool = True, device_index: int = 0, np_threads: int = 8, torch_hub_dir: str = model_dir) -> vs.VideoNode:
"""A Deep Learning based project for colorizing and restoring old images and video using Deoldify and DDColor
:param clip: clip to process, only RGB24 format is supported.
:param color_method: method used to combine deoldify() with ddcolor() (default = 2):
0 : deoldify only (no merge)
1 : ddcolor only (no merge)
2 : Simple Merge:
the images are combined using a weighted merge, where the parameter merge_weight
represent the weight assigned to the colors provided by ddcolor()
3 : Adaptive Luma Merge:
given that the ddcolor() perfomance is quite bad on dark scenes, the images are
combined by decreasing the weight assigned to ddcolor() when the luma is
below a given threshold given by: luma_threshold.
For example with: luma_threshold = 0.6 and alpha = 1, the weight assigned to
ddcolor() will start to decrease linearly when the luma < 60% till "min_weight".
For alpha=2, begins to decrease quadratically.
4 : Constrained Chroma Merge:
given that the colors provided by deoldify() are more conservative and stable
than the colors obtained with ddcolor() images are combined by assigning
a limit to the amount of difference in chroma values between deoldify() and
ddcolor() this limit is defined by the parameter threshold. The limit is applied
to the image converted to "YUV". For example when threshold=0.1, the chroma
values "U","V" of ddcolor() image will be constrained to have an absolute
percentage difference respect to "U","V" provided by deoldify() not higher than 10%
5 : Luma Masked Merge:
the clips are combined using a mask merge, the pixels of ddcolor() with luma < luma_mask_limit
will be filled with the pixels of deoldify(), if the parameter merge_weight > 0
the final masked image will be merged again with deoldify()
:param merge_weight: weight given to ddcolor's clip in all merge methods, range [0-1] (0.01=1%)
:param deoldify_params: parameters for deoldify():
[0] deoldify model to use (default = 0):
0 = ColorizeVideo_gen
1 = ColorizeStable_gen
2 = ColorizeArtistic_gen
[1] render factor for the model, range: 10-44 (default = 24).
[2] saturation parameter to apply to deoldify color model (default = 1)
[3] hue parameter to apply to deoldify color model (default = 1)
:param ddcolor_params: parameters for ddcolor():
[0] ddcolor model (default = 1):
0 = ddcolor_modelscope,
1 = ddcolor_artistic
[1] render factor for the model, if = 0 will be auto selected
(default = 24) [range: 0, 10-64]
[2] saturation parameter to apply to deoldify color model (default = 1)
[3] hue parameter to apply to deoldify color model (default = 1)
[4] enable FP16 inference in DDColor
:param ddcolor_tweaks: tweak parameters for ddcolor():
[0] : luma_constrained_tweak -> luma constrained ddcolor preprocess enabled (default = False), range: [True, False]
when enaabled the average luma of a video clip will be forced to don't be below the value
defined by the parameter "luma_min". The function allow to modify the gamma
of the clip if the average luma is below the parameter "gamma_luma_min"
[1] : luma_min -> luma (%) min value for tweak activation (default = 0, non activation), range [0-1]
[2] : gamma_luma_min -> luma (%) min value for gamma tweak activation (default = 0, non activation), range [0-1]
[3] : ddcolor tweak's bright (default = 0)
[4] : ddcolor tweak's constrast (default = 1)
[5] : ddcolor tweak's gamma (default = 1)
[6] : enable/disable FP16 in ddcolor inference
[7] : ddcolor's number of CUDA streams to enqueue the kernels (default = 1) (default = 1)
:param combine_params: list with the parameters used by combining methods:
[0] : if true invert the clip order in all the combining methods
-parameters for method: "Adaptive Luma Merge" (method=3)
[1] : luma_threshold, used by: AdaptiveLumaMerge, range [0-1] (0.01=1%)
[2] : alpha (float), used by: AdaptiveLumaMerge, range [>0]
[3] : min_weight, used by: AdaptiveLumaMerge, range [0-1] (0.01=1%)
-parameters for method: "Constrained Chroma Merge" (method=4)
[4] : chroma_threshold (%), used by: ConstrainedChromaMerge range [0-1] (0.01=1%)
-parameters for method: "Luma Masked Merge" (method=5)
[5] : luma_mask_limit, luma limit for build the mask used in Luma Masked Merge, range [0-1] (0.01=1%)
[6] : luma_white_limit, if > luma_mask_limit will be applied a gradient till luma_white_limit, range [0-1] (0.01=1%)
[7] : luma_mask_sat: if < 1 the ddcolor dark pixels will substitute with the desaturated deoldify pixels, range [0-1] (0.01=1%)
:param darkness: parameters for darken the clip's dark portions, which sometimes are wrongly colored by the color models\
[0] : darkness_enabled (bool), if true the filter is enabled
[1] : dark_threshold, luma threshold to select the dark area, range [0.1-0.35] (0.01=1%)
[3] : dark_amount: amount of desaturation to apply to the dark area, range [0-1]
:param chroma_resize: if True will be enabled the chroma_resize: the cololorization will be applied to a clip with the same
size used for the models inference(), but the final resolution will be the one of the original clip.
:param device_index: device ordinal of the GPU, choices: GPU0...GPU7, CPU=99 (default = 0)
:param np_threads: number of threads used by numpy, range: 1-32 (default = 8)
:param torch_hub_dir: torch hub dir location, default is model directory,
if set to None will switch to torch cache dir.
"""
As you can see the main changes applied are:
1) The main parameters of deoldify/ddcolor are now grouped in separated lists.
2) All the tweaks parameters of ddcolor (including the fp16 flag) has been included in a dedicated list.
3) All the parameters used by the combination methods are also grouped in a dedicated list
4) "dd_mthod" has been renamed in "color_method"
5) the filter "darkness" now has really the meaning to dark more the dark areas, while the "chroma smoothing" version has been moved in ddeoldify_stabilizer
The header for ddeoldify_stabilizer is
def ddeoldify_stabilizer(clip: vs.VideoNode, chroma_smoothing: list = [False, 0.1, 0.2, 0.5, 0.8], chroma_stabilizer: list = [True, 5, 'arithmetic', 1, 60, 0.5, 0.8, 0], chroma_limiter: list = [False, 0.02], chroma_resize: list = [True, 24]) -> vs.VideoNode:
"""Video color stabilization filter, which can be applied to stabilize the chroma components in ddeoldify colored clips.
:param clip: clip to process, only RGB24 format is supported.
:param chroma_smoothing: parameters to adjust the saturation and "vibrancy" of the clip.
[0] : chroma_smoothing_enabled (bool), if true the filter is enabled
[1] : dark_threshold, luma threshold to select the dark area, range [0-1] (0.01=1%)
[2] : white_threshold, if > dark_threshold will be applied a gradient till white_threshold, range [0-1] (0.01=1%)
[3] : dark_sat: amount of desaturation to apply to the dark area, range [0-1]
[4] : dark_bright (float): darkness parameter it used to reduce the "V" component in "HSV" colorspace, range [0, 1]
:param chroma_stabilizer: parameters for the temporal color stabilizer
[0] : colstab_enabled (bool), if true the filter will be enabled
[1] : colstab_nframes, number of frames to be used in the stabilizer, range[3-15]
[2] : colstab_mode, type of average used by the stabilizer: range['arithmetic', 'weighted']
[3] : colstab_sat: saturation applied to the restored gray prixels [0,1]
[4] : colstab_tht, threshold to detect gray pixels, range [0,235], if=0 is not applied the restore
[5] : colstab_weight, weight to blend the restored imaage, range [0,1], if=0 is not applied the blending
[6] : colstab_tht_scen, threshold for scene change detection (default = 0.8), range [0,1]
[7] : colstab_algo, algo used for stabilization: 0=clips (default), 1=frames. It is an
internal parameter added for internal testing, algo=0 is 40%-60% faster and should be always used.
:param chroma_limiter: parameters for the temporal color limiter
[0] : colimit_enabled (bool), if true the filter will be applied after the merge of Deoldify and DDColor
[1] : colimit_deviation, chroma of current frame will be forced to be inside the range defined by "deviation", range[0.01-0.5]
:param chroma_resize: parameters for the chroma resizer pre-filter
[0] : if True the chroma resizer is enabled (can improve the filters speed significantly)
[1] : render_factor to apply to the filters, the frame size will be the same used for the models inference(),
but the final resolution will be the one of the original clip.
In the filter chroma_stabilizer the last parameter algo should not be included in the GUI since has been added for internal testing.
This filter is the only enabled by default, because is the more effective in stabilizing the colors
I hope that you will be able to apply the changes in Hybrid.
Thanks and Happy Easter,
Dan
I small update (no changes on filters headers)
You manage to add more and more parameters.
ddeoldify has 33 parameters (not counting the 'clip' an 'torch_hub_dir' parameters)
ddeoldify_stabilizer has 16 parameters (not counting the 'clip' and 'colstab_algo' parameters)
Nearly 50 parameters, that is not usable or maintainable.
=> Sorry, but I'll have to think about whether:
a. I drop support for the filter altogether
or
b. how I can remove options from the UI and add a custom command line for the user.
Regarding a., Atm. I think I will adjust Hybrid to only, set:
color_method
merge_weight
deoldify_params
ddcolor_params
device_index
np_threads
and add a single custom command line addition.
Main downside to this, aside from that the user will have to read up on how to use this filter is that Hybrid can no longer call dd_video_stabilizer/ddeoldify_stabilizer.
The user would have to do that on their own in a custom section. (which probably nobody nearly would do)
=> not sure whether dropping support for the filter might be a better option.
:param ddcolor_params: parameters for ddcolor():
[0] ddcolor model (default = 1):
0 = ddcolor_modelscope,
1 = ddcolor_artistic
[1] render factor for the model, if = 0 will be auto selected
(default = 24) [range: 0, 10-64]
[2] saturation parameter to apply to deoldify color model (default = 1)
[3] hue parameter to apply to deoldify color model (default = 1)
[4] enable FP16 inference in DDColor
:param ddcolor_tweaks: tweak parameters for ddcolor():
[0] : luma_constrained_tweak -> luma constrained ddcolor preprocess enabled (default = False), range: [True, False]
when enaabled the average luma of a video clip will be forced to don't be below the value
defined by the parameter "luma_min". The function allow to modify the gamma
of the clip if the average luma is below the parameter "gamma_luma_min"
[1] : luma_min -> luma (%) min value for tweak activation (default = 0, non activation), range [0-1]
[2] : gamma_luma_min -> luma (%) min value for gamma tweak activation (default = 0, non activation), range [0-1]
[3] : ddcolor tweak's bright (default = 0)
[4] : ddcolor tweak's constrast (default = 1)
[5] : ddcolor tweak's gamma (default = 1)
[6] : enable/disable FP16 in ddcolor inference
[7] : ddcolor's number of CUDA streams to enqueue the kernels (default = 1) (default = 1)
What's the difference between ddcolor_params[4] and ddcolor_tweaks[6] ?
(29.03.2024, 16:53)Selur Wrote: What's the difference between ddcolor_params[4] and ddcolor_tweaks[6] ?
Cu Selur
Previously I added the fps16 flag in the ddcolor list, then I decided to move this parameter in ddcolor_tweak and I forget to remove it from the description of ddcolor list.
If you prefer I can add in the ddcolor list.
In the ddcolor_tweak list the most useful params are: bright, gamma, fp16, you can drop the support for the other options.
If you make a proposal of what you can keep, I will change the filter header.
I created a new dev version and converted most of the lists to LineEdit-Fields.
This way the logic in the UI is simplified a lot.
Dev version is available under the old link. (includes the .dlls&co since I'm switching to Qt 6.6.3)
=> UI is way less user friendly for the filter this way, but given that the filter has ~50 parameters and probably will get more in the future, that goal died a while ago.
I like your proposal, having grouped the parameters with the same scope helped going in this direction.
I have only the following observations:
1) the lists for filters "Chroma smoothing/stabilizer" start with a boolean flag to enable them.
When the list is checked/unchecked can you automatically change the flag to True/False ?
2) Can you dedicate to the parameter "Darkness" a dedicated GUI params like "Chroma Limiter" ?
