Deoldify Vapoursynth filter

[Dan64]

Hello Selur,

  I finally released the new version 4.5.0 on github: https://github.com/dan64/vs-deoldify/rel...tag/v4.5.0
  Respect to the last RC version there are the following changes:

  HAVC_main:

     - in DeepExMethod has been added the method: 5
     - DeepExVivid: was added (before was used a constant value of True)

  HAVC_deepex:

    - method: has been added the method 5

  In the GUI the parameters render_speed and render_vivid can be set only for DeepEX. 
  This behaviour should be changed to allow also ColorMNet to update these parameters.

  I hope that you have time to include in Hybrid this version.

Thanks,
Dan

[Selur]

see my last post (no gui adjustments there so far), last version does not work for me.

[Selur]

In the GUI the parameters render_speed and render_vivid can be set only for DeepEX.
This behaviour should be changed to allow also ColorMNet to update these parameters.

I adjusted the test version.

DeepExVivid: was added (before was used a constant value of True)

Assuming it has no side effects, I added the value.

method: has been added the method 5

should be fine with the above adjustment.

=> updated deoldify test download
Any idea regarding 'Failed to initialize ColorMNet[remote] try ColorMNet[local]' ?

Cu Selur

[Dan64]

I tested the new version and it seems working.

The ColorMNet[local] is working on your side ?

In my experience the only case that i found for failing in initialize ColorMNet[remote] is the issue of some warning (even in some loaded library) that are not trapped by the Vapoursynth logger because the "remote" thread don't install any logger. This is the reason why I tried to suppress all warnings in __init__.py.

I don't spent too many time to fix this issue because the previous version was working on your side.
I will try to reproduce the problem, to see if there is a way to fix it.

In any case in the recommended settings for Hybrid: https://github.com/dan64/vs-deoldify#col...ing-hybrid
I suggested to use ColorMNet[local], because I recommended to set SCMinFreq=3 and in this case, since Vivid is also checked, the memory will be reset every 3 frames that is below the limit for using ColorMNet[local]. To feed the frames to ColorMNet[remote] they need to be converted in ASCII and then after the coloring need to be converted again in ASCII when the colored frames are retuned back to the client. This conversion is introducing an overhead that is not justified in this case because the memory is reset every 3 frames.

I hope to be able to fix the issue, in meanwhile you can use ColorMNet[local].

Thanks,
Dan

[Selur]

I have no clue about 'ColorMNet[local]', no clue what to change or test.

I don't spent too many time to fix this issue because the previous version was working on your side.

It did not work with the current Vapoursynth R70.

=> unless it works on my system, I will not make a public release of this.

in meanwhile you can use ColorMNet[local].

What? How?

[Selur]

Using

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExMethod=0, DeepExRefMerge=0, ScFrameDir=None, DeepExModel=0, DeepExEncMode=1, DeepExMaxMemFrames=0)

it fails with:

Failed to evaluate the script:
Python exception: DLL load failed while importing spatial_correlation_sampler_backend: Die angegebene Prozedur wurde nicht gefunden.

Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3387, in vapoursynth._vpy_evaluate
File "src\\cython\\vapoursynth.pyx", line 3388, in vapoursynth._vpy_evaluate
File "J:\tmp\tempPreviewVapoursynthFile18_01_34_357.vpy", line 46, in
clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExMethod=0, DeepExRefMerge=0, ScFrameDir=None, DeepExModel=0, DeepExEncMode=1, DeepExMaxMemFrames=0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 317, in HAVC_main
clip_colored = HAVC_deepex(clip=clip, clip_ref=clip_ref, method=DeepExMethod, render_speed=DeepExPreset,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 615, in HAVC_deepex
clip_colored = vs_colormnet(clip, clip_ref, clip_sc, image_size=-1, enable_resize=enable_resize,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\vsslib\vsmodels.py", line 48, in vs_colormnet
return vs_colormnet_local(clip, clip_ref, clip_sc, image_size, enable_resize, frame_propagate,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 52, in vs_colormnet_local
colorizer = ColorMNetRender(image_size=image_size, vid_length=vid_length, enable_resize=enable_resize,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 90, in __init__
self._colorize_init(image_size, vid_length, propagate)
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 144, in _colorize_init
self.network = ColorMNet(self.config, self.config['model']).cuda().eval()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\model\network.py", line 37, in __init__
self.short_term_attn = LocalGatedPropagation(d_qk=64, # 256
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\model\attention.py", line 763, in __init__
from spatial_correlation_sampler import SpatialCorrelationSampler
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\spatial_correlation_sampler\__init__.py", line 1, in
from .spatial_correlation_sampler import SpatialCorrelationSampler, spatial_correlation_sample
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\spatial_correlation_sampler\spatial_correlation_sampler.py", line 6, in
import spatial_correlation_sampler_backend as correlation
ImportError: DLL load failed while importing spatial_correlation_sampler_backend: Die angegebene Prozedur wurde nicht gefunden.

So neither 'remote' nor 'local' work.

Cu Selur

Ps.: did small gui adjustment in the deoldify test version and refreshed the download.

[Dan64]

Now I understood,  it is the same problem previously reported.

This is not strictly related to R70, because as you stated in post: #718

using the working old setup, copying R70 into it and installing the Vapoursynth whl file and the ColorMNet is still working.

The problem is related to the new pytorch version and mybe to the new CUDA 12.4 as stated in: #719

Where I can found your R70 testing version ?

Dan

[Selur]

Current Hybrid already comes with R70, but you can get the latest dev and addons in the 'experimental' folder of my GoogleDrive, which is linked on the download page. Simply installing the dev, extracting the torch-addon, replacing Hybrid.exe with the deoldify test version and installing deoldify should show the problem.

Cu Selur

[Dan64]

Current Hybrid already comes with R70, but you can get the latest dev and addons in the 'experimental' folder of my GoogleDrive, which is linked on the download page. Simply installing the dev, extracting the torch-addon, replacing Hybrid.exe with the deoldify test version and installing deoldify should show the problem.

I followed your steps, and I get the same error regarding spatial_correlation_sampler_backend
But I get also the following message.

Version mismatch: The VapourSynth Python module version is R70 but the VapourSynth core library is R68. This usually indicates a broken install.

This message is displayed only If I enable ColorMNet, If I use only HAVC this message is not displayed, very strange...

Dan

[Selur]

Not getting that here.
You did use the torch-addon from the experimental folder, right?
I only get:
(remote)

2024-11-10 06:19:51.295
Failed to evaluate the script:
Python exception: Failed to initialize ColorMNet[remote] try ColorMNet[local]

Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3387, in vapoursynth._vpy_evaluate
File "src\\cython\\vapoursynth.pyx", line 3388, in vapoursynth._vpy_evaluate
File "J:\tmp\tempPreviewVapoursynthFile06_19_36_172.vpy", line 44, in
clip = HAVC_main(clip=clip, ColorFix="none", EnableDeepEx=True, DeepExMethod=0, DeepExRefMerge=0, ScFrameDir=None, DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 317, in HAVC_main
clip_colored = HAVC_deepex(clip=clip, clip_ref=clip_ref, method=DeepExMethod, render_speed=DeepExPreset,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 615, in HAVC_deepex
clip_colored = vs_colormnet(clip, clip_ref, clip_sc, image_size=-1, enable_resize=enable_resize,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\vsslib\vsmodels.py", line 45, in vs_colormnet
return vs_colormnet_remote(clip, clip_ref, clip_sc, image_size, enable_resize, frame_propagate,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 136, in vs_colormnet_remote
HAVC_LogMessage(MessageType.EXCEPTION, "Failed to initialize ColorMNet[remote] try ColorMNet[local]")
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\vsslib\vsutils.py", line 47, in HAVC_LogMessage
raise vs.Error(message_text)
vapoursynth.Error: Failed to initialize ColorMNet[remote] try ColorMNet[local]

and:
(local)

Failed to evaluate the script:
Python exception: DLL load failed while importing spatial_correlation_sampler_backend: Die angegebene Prozedur wurde nicht gefunden.

Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3387, in vapoursynth._vpy_evaluate
File "src\\cython\\vapoursynth.pyx", line 3388, in vapoursynth._vpy_evaluate
File "J:\tmp\tempPreviewVapoursynthFile06_27_29_480.vpy", line 44, in
clip = HAVC_main(clip=clip, ColorFix="none", EnableDeepEx=True, DeepExMethod=0, DeepExRefMerge=0, ScFrameDir=None, DeepExModel=0, DeepExEncMode=1, DeepExMaxMemFrames=0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 317, in HAVC_main
clip_colored = HAVC_deepex(clip=clip, clip_ref=clip_ref, method=DeepExMethod, render_speed=DeepExPreset,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 615, in HAVC_deepex
clip_colored = vs_colormnet(clip, clip_ref, clip_sc, image_size=-1, enable_resize=enable_resize,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\vsslib\vsmodels.py", line 48, in vs_colormnet
return vs_colormnet_local(clip, clip_ref, clip_sc, image_size, enable_resize, frame_propagate,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 52, in vs_colormnet_local
colorizer = ColorMNetRender(image_size=image_size, vid_length=vid_length, enable_resize=enable_resize,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 90, in __init__
self._colorize_init(image_size, vid_length, propagate)
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 144, in _colorize_init
self.network = ColorMNet(self.config, self.config['model']).cuda().eval()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\model\network.py", line 37, in __init__
self.short_term_attn = LocalGatedPropagation(d_qk=64, # 256
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\model\attention.py", line 763, in __init__
from spatial_correlation_sampler import SpatialCorrelationSampler
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\spatial_correlation_sampler\__init__.py", line 1, in
from .spatial_correlation_sampler import SpatialCorrelationSampler, spatial_correlation_sample
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\spatial_correlation_sampler\spatial_correlation_sampler.py", line 6, in
import spatial_correlation_sampler_backend as correlation
ImportError: DLL load failed while importing spatial_correlation_sampler_backend: Die angegebene Prozedur wurde nicht gefunden.

(with both vsViewer and 'vspipe --info')

Cu Selur