Deoldify Vapoursynth filter

[Selur]

I emptied the debug output path ('Config->Paths->Debug output path'), enabled the debug output creation and the debug output was properly created in the output folder.

In my configuration I set to adjust the path with output folder, but this adjustment do not apply any more on debug output.

If 'Config->Paths->Debug output path' is empty, it should work as before, since Hybrid should fall back to using the output path as debug output path.

If you set 'Config->Paths->Debug output path' it is independent of the output path.
=> atm. I don't see that there is anything to fix.

Please remove them because I added the controls in HAVC_main.

Updated a deoldify test version, to a version without the checks.
Wouldn't it make sense to keep these checks in the GUI too, to avoid having someone enable some DeOldify setting without checking the preview and then crash using processing?

Cu Selur

[Selur]

btw. small heads-up: next public release of Hybrid will use Vapoursynth R70.
(only thing so far that I can't get to run with it is 'vsgmfss-fortuna' which I personally don't mind since I never found any real use for it. I breaks dependencies due to torch-tensorrt-fx-only.)

Cu Selur

[Dan64]

Updated a deoldify test version, to a version without the checks.
Wouldn't it make sense to keep these checks in the GUI too, to avoid having someone enable some DeOldify setting without checking the preview and then crash using processing?

Released new RC15, in this release are applied the following changes:

1) the input errors now are reported as FATAL error, the process is stopped, but are not shown all the function calls as before, only the error message, this should be less confusing  
2) now in "sync" mode the the progress of coloring is reported as: "Colored Frame: n/total_frames" 

In vsPipe there is the following code:

static const char *messageTypeToString(int msgType) {
    switch (msgType) {
        case mtDebug: return "Debug";
        case mtInformation: return "Information";
        case mtWarning: return "Warning";
        case mtCritical: return "Critical";
        case mtFatal: return "Fatal";
        default: return "";
    }
}

Since I changed the message type to INFORMATION, vsPipe will output: Infomation :  Colored Frame: n/total_frames 

In HybridDebugOutput the message regarding the progress is reported only if I use the message type FATAL, but since the process will be stopped, it cannot be used (I attached the file).

Dan

[Selur]

Will try to look at it tomorrow evening, if that doesn't work the day after tomorrow after work.

[Selur]

Using:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'F:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# loading plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/BestSource/BestSource.dll")
# Import scripts
import validate
# Source: 'G:\TestClips&Co\files\test.avi'
# Current color space: YUV420P8, bit depth: 8, resolution: 640x352, frame rate: 25fps, scanorder: progressive, yuv luminance scale: limited, matrix: 470bg, format: MPEG-4 Visual
# Loading G:\TestClips&Co\files\test.avi using BestSource)
clip = core.bs.VideoSource(source="G:/TestClips&Co/files/test.avi", cachepath="J:/tmp/2024-09-24@16_52_47_9210/test.json", track=0, hwdevice="cuda")
frame = clip.get_frame(0)
# setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT470_BG)
# setting color transfer (vs.TRANSFER_BT601), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT601)
# setting color primaries info (to vs.PRIMARIES_BT470_BG), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT470_BG)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED)
# making sure frame rate is set to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive
# changing color matrix from '470bg' to '709' for vsDeOldify
clip = core.resize.Bicubic(clip, matrix_in_s="470bg", matrix_s="709")
# changing range from limited to full range for vsDeOldify
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_FULL)
# adjusting color space from YUV420P8 to RGB24 for vsDeOldify
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="709", range_s="full")
# adding colors using DeOldify
from vsdeoldify import HAVC_main
clip = HAVC_main(clip=clip, ColorFix="none", EnableDeepEx=True, DeepExMethod=0, DeepExRefMerge=0, ScFrameDir=None, ScThreshold=0.00, DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)
# internally changing color matrix for YUV<>RGB to '470bg' undoing color matrix change for vsDeOldify
# changing range from full to limited range for vsDeOldify
clip = core.resize.Bicubic(clip, range_in_s="full", range_s="limited")
# adjusting output color from: RGB24 to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_s="limited")
# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# output
clip.set_output()

and:

"F:\Hybrid\64bit\Vapoursynth\vspipe.exe" "J:\tmp\2024-09-24@16_52_47_9210\encodingTempSynthSkript_2024-09-24@16_52_47_9210_0.vpy" NUL -c y4m

to test the output,
I get:

Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\kornia\feature\lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)

Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\deepex\models\vgg19_gray.py:130: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  model.load_state_dict(torch.load(vgg19_gray_path))

Script evaluation failed:
Python exception: ddcolor() got an unexpected keyword argument 'scenechange'

Traceback (most recent call last):
  File "src\\cython\\vapoursynth.pyx", line 3387, in vapoursynth._vpy_evaluate
  File "src\\cython\\vapoursynth.pyx", line 3388, in vapoursynth._vpy_evaluate
  File "J:\tmp\2024-09-24@16_52_47_9210\encodingTempSynthSkript_2024-09-24@16_52_47_9210_0.vpy", line 44, in <module>
    clip = HAVC_main(clip=clip, ColorFix="none", EnableDeepEx=True, DeepExMethod=0, DeepExRefMerge=0, ScFrameDir=None, ScThreshold=0.00, DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 288, in HAVC_main
    clip_ref = HAVC_ddeoldify(clip, method=2, mweight=ddcolor_weight[w_id],
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 769, in HAVC_ddeoldify
    clipb = vs_sc_ddcolor(clip, method=method, model=ddcolor_model, render_factor=ddcolor_rf, tweaks_enabled=ddtweak,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\vsslib\vsmodels.py", line 241, in vs_sc_ddcolor
    clipb = vsddcolor.ddcolor(clipb.resize.Bicubic(format=vs.RGBH, range_s="full"), model=model,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
TypeError: ddcolor() got an unexpected keyword argument 'scenechange'

Cu Selur

Ps.: using vsddcolor-1.0.1-py3-none-any.whl.zip from #468

[Dan64]

The correct call is:

clip = HAVC_main(clip=clip, ColorFix="none", EnableDeepEx=True, DeepExMethod=0, DeepExRefMerge=0, ScFrameDir=None, ScThreshold=0.05, DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)

But my check is postponed when is called the HAVC_deepx. I will add the check also in HAVC_main.

In any case the error in ddcolor is strange, on my PC this error is not triggered.

Please check if in Lib\site-packages\vsddcolor\__init__.py there is the following header

"""
-------------------------------------------------------------------------------
Author: HolyWu
Date: 2023-12-13
version:
LastEditors: Dan64
LastEditTime: 2024-05-20
-------------------------------------------------------------------------------
Description: Added optional parameter "scenechange" if "True" will be colored
only the frames with property '_SceneChangePrev' == 1
-------------------------------------------------------------------------------
main Vapoursynth wrapper for model:  "DDColor"
URL: https://github.com/HolyWu/vs-ddcolor
"""

Dan

[Selur]

'_SceneChangePrev' is set by:

clip = core.misc.SCDetect(clip=clip,threshold=THRESHOLD_VALUE)

=> did you remove that somewhere? You must have called it previously

[Dan64]

yes, it is called in HAVC_ddeoldify(), in this piece of code

scenechange = not (sc_threshold == 0 and sc_min_freq == 0)

    clip = SceneDetect(clip, threshold=sc_threshold, frequency=sc_min_freq)

    frame_size = min(max(ddcolor_rf, deoldify_rf) * 16, clip.width)  # frame size calculation for inference()
    clip_orig = clip;
    clip = clip.resize.Spline64(width=frame_size, height=frame_size)

    clipa = vs_sc_deoldify(clip, method=method, model=deoldify_model, render_factor=deoldify_rf,
                           scenechange=scenechange, package_dir=package_dir)
    clipb = vs_sc_ddcolor(clip, method=method, model=ddcolor_model, render_factor=ddcolor_rf, tweaks_enabled=ddtweak,
                          tweaks=ddtweak_p, enable_fp16=ddcolor_enable_fp16, scenechange=scenechange,
                          device_index=device_index)

I included a new RC16 where I anticipated the check in HAVC_main().

Dan

[Selur]

Okay, something is wrong with my Lib\site-packages.
Now I get:

2024-09-24 21:00:13.239
F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\kornia\feature\lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
@torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)

F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\kornia\feature\lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
@torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)

F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\deepex\models\vgg19_gray.py:130: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
model.load_state_dict(torch.load(vgg19_gray_path))

F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\deepex\models\vgg19_gray.py:130: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
model.load_state_dict(torch.load(vgg19_gray_path))

2024-09-24 21:00:26.437
Failed to evaluate the script:
Python exception: DLL load failed while importing spatial_correlation_sampler_backend: Die angegebene Prozedur wurde nicht gefunden.

Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3387, in vapoursynth._vpy_evaluate
File "src\\cython\\vapoursynth.pyx", line 3388, in vapoursynth._vpy_evaluate
File "J:\tmp\tempPreviewVapoursynthFile21_00_10_352.vpy", line 45, in
clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExMethod=0, DeepExRefMerge=0, ScFrameDir=None, DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 297, in HAVC_main
clip_colored = HAVC_deepex(clip=clip, clip_ref=clip_ref, method=DeepExMethod, render_speed=DeepExPreset,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 574, in HAVC_deepex
clip_colored = vs_colormnet(clip, clip_ref, image_size=-1, enable_resize=enable_resize,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\vsslib\vsmodels.py", line 41, in vs_colormnet
return vs_colormnet_sync(clip, clip_ref, image_size, enable_resize, frame_propagate, max_memory_frames)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 177, in vs_colormnet_sync
vid_render = colormnet_colorizer(image_size=image_size, vid_length=vid_length, enable_resize=enable_resize,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 45, in colormnet_colorizer
return ColorMNetRender(image_size=image_size, vid_length=vid_length, enable_resize=enable_resize,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 83, in __init__
self._colorize_init(image_size, vid_length, propagate)
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 137, in _colorize_init
self.network = ColorMNet(self.config, self.config['model']).cuda().eval()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\model\network.py", line 37, in __init__
self.short_term_attn = LocalGatedPropagation(d_qk=64, # 256
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\model\attention.py", line 763, in __init__
from spatial_correlation_sampler import SpatialCorrelationSampler
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\spatial_correlation_sampler\__init__.py", line 1, in
from .spatial_correlation_sampler import SpatialCorrelationSampler, spatial_correlation_sample
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\spatial_correlation_sampler\spatial_correlation_sampler.py", line 6, in
import spatial_correlation_sampler_backend as correlation
ImportError: DLL load failed while importing spatial_correlation_sampler_backend: Die angegebene Prozedur wurde nicht gefunden.

I did extract into spatial_correlation_sampler-0.5.0-py312-cp312-win_amd64.whl.zip under "Hybrid\64bit\Vapoursynth\Lib\site-packages" but it seems like I'm missing some dll.
(going to bed now)

Cu Selur

[Dan64]

But I don't understand what was changed.
In the previous post you was able to run some test successfully.
spatial_correlation_sampler is necessary for ColorMNet, if you was able to run it, this implies that was properly installed.

Did you changed something ?

To work this filter, you should find in Lib\site-packages
the files: spatial_correlation_sampler_backend.cp312-win_amd64.pyd, spatial_correlation_sampler_backend.py
and the folders: spatial_correlation_sampler, spatial_correlation_sampler-0.5.0.dist-info
moreover in  Lib\site-packages\__pycache__
you should find spatial_correlation_sampler_backend.cpython-312.pyc

please check.

Dan