Selur's Little Message Board - Deoldify Vapoursynth filter

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77

I emptied the debug output path ('Config->Paths->Debug output path'), enabled the debug output creation and the debug output was properly created in the output folder.

Quote: In my configuration I set to adjust the path with output folder, but this adjustment do not apply any more on debug output.

If 'Config->Paths->Debug output path' is empty, it should work as before, since Hybrid should fall back to using the output path as debug output path.

If you set 'Config->Paths->Debug output path' it is independent of the output path.
=> atm. I don't see that there is anything to fix. Smile

Quote:Please remove them because I added the controls in HAVC_main.

Updated a deoldify test version, to a version without the checks.
Wouldn't it make sense to keep these checks in the GUI too, to avoid having someone enable some DeOldify setting without checking the preview and then crash using processing?

Cu Selur

btw. small heads-up: next public release of Hybrid will use Vapoursynth R70.
(only thing so far that I can't get to run with it is 'vsgmfss-fortuna' which I personally don't mind since I never found any real use for it. I breaks dependencies due to torch-tensorrt-fx-only.)

Cu Selur

(23.09.2024, 05:16)Selur Wrote: [ -> ]Updated a deoldify test version, to a version without the checks.
Wouldn't it make sense to keep these checks in the GUI too, to avoid having someone enable some DeOldify setting without checking the preview and then crash using processing?

Released new RC15, in this release are applied the following changes:

1) the input errors now are reported as FATAL error, the process is stopped, but are not shown all the function calls as before, only the error message, this should be less confusing
2) now in "sync" mode the the progress of coloring is reported as: "Colored Frame: n/total_frames"

In vsPipe there is the following code:

Code:
static const char *messageTypeToString(int msgType) {

    switch (msgType) {

        case mtDebug: return "Debug";

        case mtInformation: return "Information";

        case mtWarning: return "Warning";

        case mtCritical: return "Critical";

        case mtFatal: return "Fatal";

        default: return "";

    }

}

Since I changed the message type to INFORMATION, vsPipe will output: Infomation : Colored Frame: n/total_frames

In HybridDebugOutput the message regarding the progress is reported only if I use the message type FATAL, but since the process will be stopped, it cannot be used (I attached the file).

Dan

Will try to look at it tomorrow evening, if that doesn't work the day after tomorrow after work.

Using:

Code:
# Imports

import vapoursynth as vs

# getting Vapoursynth core

import sys

import os

core = vs.core

# Import scripts folder

scriptPath = 'F:/Hybrid/64bit/vsscripts'

sys.path.insert(0, os.path.abspath(scriptPath))

# loading plugins

core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")

core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/BestSource/BestSource.dll")

# Import scripts

import validate

# Source: 'G:\TestClips&Co\files\test.avi'

# Current color space: YUV420P8, bit depth: 8, resolution: 640x352, frame rate: 25fps, scanorder: progressive, yuv luminance scale: limited, matrix: 470bg, format: MPEG-4 Visual

# Loading G:\TestClips&Co\files\test.avi using BestSource)

clip = core.bs.VideoSource(source="G:/TestClips&Co/files/test.avi", cachepath="J:/tmp/2024-09-24@16_52_47_9210/test.json", track=0, hwdevice="cuda")

frame = clip.get_frame(0)

# setting color matrix to 470bg.

clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT470_BG)

# setting color transfer (vs.TRANSFER_BT601), if it is not set.

if validate.transferIsInvalid(clip):

  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT601)

# setting color primaries info (to vs.PRIMARIES_BT470_BG), if it is not set.

if validate.primariesIsInvalid(clip):

  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT470_BG)

# setting color range to TV (limited) range.

clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED)

# making sure frame rate is set to 25fps

clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)

# making sure the detected scan type is set (detected: progressive)

clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive

# changing color matrix from '470bg' to '709' for vsDeOldify

clip = core.resize.Bicubic(clip, matrix_in_s="470bg", matrix_s="709")

# changing range from limited to full range for vsDeOldify

clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")

# setting color range to PC (full) range.

clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_FULL)

# adjusting color space from YUV420P8 to RGB24 for vsDeOldify

clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="709", range_s="full")

# adding colors using DeOldify

from vsdeoldify import HAVC_main

clip = HAVC_main(clip=clip, ColorFix="none", EnableDeepEx=True, DeepExMethod=0, DeepExRefMerge=0, ScFrameDir=None, ScThreshold=0.00, DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)

# internally changing color matrix for YUV<>RGB to '470bg' undoing color matrix change for vsDeOldify

# changing range from full to limited range for vsDeOldify

clip = core.resize.Bicubic(clip, range_in_s="full", range_s="limited")

# adjusting output color from: RGB24 to YUV420P10 for NVEncModel

clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_s="limited")

# set output frame rate to 25fps (progressive)

clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)

# output

clip.set_output()

and:

Code:
"F:\Hybrid\64bit\Vapoursynth\vspipe.exe" "J:\tmp\2024-09-24@16_52_47_9210\encodingTempSynthSkript_2024-09-24@16_52_47_9210_0.vpy" NUL -c y4m

to test the output,
I get:

Code:
Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\kornia\feature\lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.

  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)

Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\deepex\models\vgg19_gray.py:130: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.

  model.load_state_dict(torch.load(vgg19_gray_path))

Script evaluation failed:

Python exception: ddcolor() got an unexpected keyword argument 'scenechange'

Traceback (most recent call last):

  File "src\\cython\\vapoursynth.pyx", line 3387, in vapoursynth._vpy_evaluate

  File "src\\cython\\vapoursynth.pyx", line 3388, in vapoursynth._vpy_evaluate

  File "J:\tmp\2024-09-24@16_52_47_9210\encodingTempSynthSkript_2024-09-24@16_52_47_9210_0.vpy", line 44, in <module>

    clip = HAVC_main(clip=clip, ColorFix="none", EnableDeepEx=True, DeepExMethod=0, DeepExRefMerge=0, ScFrameDir=None, ScThreshold=0.00, DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 288, in HAVC_main

    clip_ref = HAVC_ddeoldify(clip, method=2, mweight=ddcolor_weight[w_id],

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 769, in HAVC_ddeoldify

    clipb = vs_sc_ddcolor(clip, method=method, model=ddcolor_model, render_factor=ddcolor_rf, tweaks_enabled=ddtweak,

            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\vsslib\vsmodels.py", line 241, in vs_sc_ddcolor

    clipb = vsddcolor.ddcolor(clipb.resize.Bicubic(format=vs.RGBH, range_s="full"), model=model,

            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context

    return func(*args, **kwargs)

           ^^^^^^^^^^^^^^^^^^^^^

TypeError: ddcolor() got an unexpected keyword argument 'scenechange'

Cu Selur

Ps.: using vsddcolor-1.0.1-py3-none-any.whl.zip from #468

The correct call is:

Code:
clip = HAVC_main(clip=clip, ColorFix="none", EnableDeepEx=True, DeepExMethod=0, DeepExRefMerge=0, ScFrameDir=None, ScThreshold=0.05, DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)

But my check is postponed when is called the HAVC_deepx. I will add the check also in HAVC_main.

In any case the error in ddcolor is strange, on my PC this error is not triggered.

Please check if in Lib\site-packages\vsddcolor\__init__.py there is the following header

Code:
"""

-------------------------------------------------------------------------------

Author: HolyWu

Date: 2023-12-13

version:

LastEditors: Dan64

LastEditTime: 2024-05-20

-------------------------------------------------------------------------------

Description: Added optional parameter "scenechange" if "True" will be colored

only the frames with property '_SceneChangePrev' == 1

-------------------------------------------------------------------------------

main Vapoursynth wrapper for model:  "DDColor"

URL: https://github.com/HolyWu/vs-ddcolor

"""

Dan

'_SceneChangePrev' is set by:

Code:
clip = core.misc.SCDetect(clip=clip,threshold=THRESHOLD_VALUE)

=> did you remove that somewhere? You must have called it previously

yes, it is called in HAVC_ddeoldify(), in this piece of code

Code:
scenechange = not (sc_threshold == 0 and sc_min_freq == 0)

    clip = SceneDetect(clip, threshold=sc_threshold, frequency=sc_min_freq)

    frame_size = min(max(ddcolor_rf, deoldify_rf) * 16, clip.width)  # frame size calculation for inference()

    clip_orig = clip;

    clip = clip.resize.Spline64(width=frame_size, height=frame_size)

    clipa = vs_sc_deoldify(clip, method=method, model=deoldify_model, render_factor=deoldify_rf,

                           scenechange=scenechange, package_dir=package_dir)

    clipb = vs_sc_ddcolor(clip, method=method, model=ddcolor_model, render_factor=ddcolor_rf, tweaks_enabled=ddtweak,

                          tweaks=ddtweak_p, enable_fp16=ddcolor_enable_fp16, scenechange=scenechange,

                          device_index=device_index)

I included a new RC16 where I anticipated the check in HAVC_main().

Dan

Okay, something is wrong with my Lib\site-packages. Sad

Now I get:

Code:
2024-09-24 21:00:13.239

F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\kornia\feature\lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.

@torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)

F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\kornia\feature\lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.

@torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)

F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\deepex\models\vgg19_gray.py:130: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.

model.load_state_dict(torch.load(vgg19_gray_path))

F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\deepex\models\vgg19_gray.py:130: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.

model.load_state_dict(torch.load(vgg19_gray_path))

2024-09-24 21:00:26.437

Failed to evaluate the script:

Python exception: DLL load failed while importing spatial_correlation_sampler_backend: Die angegebene Prozedur wurde nicht gefunden.

Traceback (most recent call last):

File "src\\cython\\vapoursynth.pyx", line 3387, in vapoursynth._vpy_evaluate

File "src\\cython\\vapoursynth.pyx", line 3388, in vapoursynth._vpy_evaluate

File "J:\tmp\tempPreviewVapoursynthFile21_00_10_352.vpy", line 45, in 

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExMethod=0, DeepExRefMerge=0, ScFrameDir=None, DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 297, in HAVC_main

clip_colored = HAVC_deepex(clip=clip, clip_ref=clip_ref, method=DeepExMethod, render_speed=DeepExPreset,

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 574, in HAVC_deepex

clip_colored = vs_colormnet(clip, clip_ref, image_size=-1, enable_resize=enable_resize,

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\vsslib\vsmodels.py", line 41, in vs_colormnet

return vs_colormnet_sync(clip, clip_ref, image_size, enable_resize, frame_propagate, max_memory_frames)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 177, in vs_colormnet_sync

vid_render = colormnet_colorizer(image_size=image_size, vid_length=vid_length, enable_resize=enable_resize,

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 45, in colormnet_colorizer

return ColorMNetRender(image_size=image_size, vid_length=vid_length, enable_resize=enable_resize,

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 83, in __init__

self._colorize_init(image_size, vid_length, propagate)

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 137, in _colorize_init

self.network = ColorMNet(self.config, self.config['model']).cuda().eval()

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\model\network.py", line 37, in __init__

self.short_term_attn = LocalGatedPropagation(d_qk=64, # 256

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\model\attention.py", line 763, in __init__

from spatial_correlation_sampler import SpatialCorrelationSampler

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\spatial_correlation_sampler\__init__.py", line 1, in 

from .spatial_correlation_sampler import SpatialCorrelationSampler, spatial_correlation_sample

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\spatial_correlation_sampler\spatial_correlation_sampler.py", line 6, in 

import spatial_correlation_sampler_backend as correlation

ImportError: DLL load failed while importing spatial_correlation_sampler_backend: Die angegebene Prozedur wurde nicht gefunden.

I did extract into spatial_correlation_sampler-0.5.0-py312-cp312-win_amd64.whl.zip under "Hybrid\64bit\Vapoursynth\Lib\site-packages" but it seems like I'm missing some dll. Sad

(going to bed now)

Cu Selur

But I don't understand what was changed.
In the previous post you was able to run some test successfully.
spatial_correlation_sampler is necessary for ColorMNet, if you was able to run it, this implies that was properly installed.

Did you changed something ?

To work this filter, you should find in Lib\site-packages
the files: spatial_correlation_sampler_backend.cp312-win_amd64.pyd, spatial_correlation_sampler_backend.py
and the folders: spatial_correlation_sampler, spatial_correlation_sampler-0.5.0.dist-info
moreover in Lib\site-packages\__pycache__
you should find spatial_correlation_sampler_backend.cpython-312.pyc

please check.

Dan