Deoldify Vapoursynth filter

[Selur]

I set up my Vapoursynth folder a new, wanting to have an overview of what needs to be installed.

To work this filter, you should find in Lib\site-packages
the files: spatial_correlation_sampler_backend.cp312-win_amd64.pyd, spatial_correlation_sampler_backend.py
and the folders: spatial_correlation_sampler, spatial_correlation_sampler-0.5.0.dist-info
moreover in Lib\site-packages\__pycache__
you should find spatial_correlation_sampler_backend.cpython-312.pyc

As far as I can see, the files are all there.
I redownloaded spatial_correlation_sampler-0.5.0-py312-cp312-win_amd64.whl.zip and extracted it into Vapoursynth/Lib/site-packages to be sure.
Do you know that dll the error is referring to?

Using my old Vapoursynth folder, it works, but in my new one something is amiss.
Uploaded the new folder to my 'experimental'-folder as 'Vapoursynth_testing-7z' no clue what is amiss.
Do you see anything missing?

Cu Selur

[Dan64]

spatial_correlation_sampler requirements are very low: requirements.txt

I compiled it against CUDA 12.1 and should work with any installation using CUDA 12.x

It is not possible to compile it inside Hybrid environment, I compiled it in colormnet environment.

The generate .egg (converted in .whl) is not installable is Hybrid, but copying the files works! (like mmvc).

I'm working to a python frame server for ColorMNet to try to solve definitely the problem regarding the memory usage and I'm quite busy atm.

Dan

[Selur]

Since BasicVSR++ is still working mmvc seems to work.

I compiled it against CUDA 12.1 and should work with any installation using CUDA 12.x

from the looks of it, I'm using: nvidia_cuda_runtime_cu12-12.6.68

I'm working to a python frame server for ColorMNet to try to solve definitely the problem regarding the memory usage and I'm quite busy atm.

No problem.

Cu Selur

[Dan64]

I tested your environment, and I get the same error:

File "D:\SYNC\Software\Hybrid\Torch\testing\Vapoursynth\Lib\site-packages\spatial_correlation_sampler\spatial_correlation_sampler.py", line 6, in <module>
    import spatial_correlation_sampler_backend as correlation
ImportError: DLL load failed while importing spatial_correlation_sampler_backend: Impossibile trovare la procedura specificata.

more precisely from spatial_correlation_sampler_backend.py that has this code

def __bootstrap__():
    global __bootstrap__, __loader__, __file__
    import sys, pkg_resources, importlib.util
    __file__ = pkg_resources.resource_filename(__name__, 'spatial_correlation_sampler_backend.cp312-win_amd64.pyd')
    __loader__ = None; del __bootstrap__, __loader__
    spec = importlib.util.spec_from_file_location(__name__,__file__)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
__bootstrap__()

'spatial_correlation_sampler_backend.cp312-win_amd64.pyd'  is the DLL not found, but is available.

try to downgrade to importlib 7.10
debugging this kind of errors in python is almost impossible.

What is strange is that on my side the DLL is found. 
I deleted the file spatial_correlation_sampler_backend.cpython-312.pyc in  Lib\site-packages\__pycache__
But the filter is working fine, it seems that this file is not necessary to load the DLL.

Don't have any idea on how to fix this problem

Dan

[Selur]

Thanks for looking into it, will try tomorrow and report.
Cu Selur

[Dan64]

In the case it could be useful I attached a comparison on the difference between my env and your testing env (obtained using WinMerge).
In the report are included only the files or folders that are different, and are a lot !

Dan

[Dan64]

Given that your new environment is very different.
My suggestion is to start with the last working environment (including python version).
Upgrade a package a time (starting from python) till you will find the incompatible package.

Dan

[hallomanbh]

2024-09-26 13:56:24.416
Script was successfully evaluated. Output video info:
Frames: 3907 | Time: 0:02:10.364 | Size: 2560x720 | FPS: 30000/1001 = 29.97 | Format: YUV420P8
2024-09-26 13:56:24.587
Qt warning: QPixmap:caled: Pixmap is a null pixmap
2024-09-26 13:56:31.554
[VSE Server]: incoming connection
[VSE Server]: ConnectedState
[VSE Server]: socket is ready to be read
[VSE Server]: connection open: true
[VSE Server]: connection readable: true
[VSE Server] - Message received: changeTo ### C:\Users\SOS\AppData\Local\Temp\tempPreviewVapoursynthFile13_56_31_542.vpy ###
2024-09-26 13:56:41.086
C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\kornia\feature\lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
@torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\kornia\feature\lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
@torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\deepex\models\vgg19_gray.py:130: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/...ted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
model.load_state_dict(torch.load(vgg19_gray_path))
C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\deepex\models\vgg19_gray.py:130: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/...ted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
model.load_state_dict(torch.load(vgg19_gray_path))
2024-09-26 13:56:45.936
Failed to evaluate the script:
Python exception: DLL load failed while importing spatial_correlation_sampler_backend: The specified procedure could not be found.
Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3387, in vapoursynth._vpy_evaluate
File "src\\cython\\vapoursynth.pyx", line 3388, in vapoursynth._vpy_evaluate
File "C:\Users\SOS\AppData\Local\Temp\tempPreviewVapoursynthFile13_56_31_542.vpy", line 43, in
clip = HAVC_deepex(clip=clip, method=3, ref_merge=0, dark=True, smooth=True, sc_framedir="C:/Users/SOS/Videos/CyberLink/ok", ex_model=0, max_memory_frames=0, encode_mode=0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 574, in HAVC_deepex
clip_colored = vs_colormnet(clip, clip_ref, image_size=-1, enable_resize=enable_resize,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\vsslib\vsmodels.py", line 37, in vs_colormnet
return vs_colormnet_batch(clip, clip_ref, image_size, enable_resize, frame_propagate, max_memory_frames)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 199, in vs_colormnet_batch
colorizer = colormnet_colorizer(image_size=image_size, vid_length=vid_length, enable_resize=enable_resize,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 45, in colormnet_colorizer
return ColorMNetRender(image_size=image_size, vid_length=vid_length, enable_resize=enable_resize,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 83, in __init__
self._colorize_init(image_size, vid_length, propagate)
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 137, in _colorize_init
self.network = ColorMNet(self.config, self.config['model']).cuda().eval()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\model\network.py", line 37, in __init__
self.short_term_attn = LocalGatedPropagation(d_qk=64, # 256
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\model\attention.py", line 763, in __init__
from spatial_correlation_sampler import SpatialCorrelationSampler
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\spatial_correlation_sampler\__init__.py", line 1, in
from .spatial_correlation_sampler import SpatialCorrelationSampler, spatial_correlation_sample
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\spatial_correlation_sampler\spatial_correlation_sampler.py", line 6, in
import spatial_correlation_sampler_backend as correlation
ImportError: DLL load failed while importing spatial_correlation_sampler_backend: The specified procedure could not be found.

[Selur]

Yes, if you are used the Vapoursynth_testing download, this is to be expected, this error was the reason for that file.

Cu Selur

[Selur]

try to downgrade to importlib 7.10

replaced the importlib folders with the older versions, but that sadly did not help.

Upgrade a package a time (starting from python) till you will find the incompatible package.

Usually I don't upgrade, I reinstall everything.
I tried a clean installation with Vapoursynth R70.

* created 'Vapoursynth'-folder
* downloaded 'Windows embeddable package (64-bit)' from https://www.python.org/downloads/release/python-3125/
* extracted it into the 'Vapoursynth'-folder
* downloaded 'VapourSynth64-Portable-R70' from https://github.com/vapoursynth/vapoursynth/releases
* extracted it into the 'Vapoursynth'-folder
* finished Vapoursynth portable installation:
  * adjusted `python312._pth` by adding:
  ```
  Scripts
  Lib\site-packages
  ```
  * installed pip:
    * download get-pip.py from https://bootstrap.pypa.io/get-pip.py into the 'Vapoursynth'-folder
    * `python get-pip.py`
  * `python -m pip install wheel\VapourSynth-70-cp312-cp312-win_amd64.whl`
* install vsrife dependencies, by calling:
  *  `python -m pip install -U packaging setuptools wheel`
  *  `python -m pip install --pre -U torch torchvision torch_tensorrt --index-url https://download.pytorch.org/whl/nightly/cu124 --extra-index-url https://pypi.nvidia.com`
* install vsrife, by calling
  * `python -m pip install -U vsrife`
  * `python -m vsrife`
* installed 'vsfemasr':
  * `python -m pip install -U vsfemasr`
  * `python -m vsfemasr` 
* installed vs-dpir, by calling
  * `python -m pip install -U vsdpir`
  * `python -m vsdpir`
* installed vsgan by calling:
  * `python -m pip install -U VSGAN`
* installed SwinIR by calling:
  * `python -m pip install -U vsswinir`
  * `python -m vsswinir`
* installed HINet by calling:
  * `python -m pip install -U vshinet`
  * `python -m vshinet`
* installed SCUNet by calling:
  * `python -m pip install -U vsscunet`
  * `python -m vsscunet`
* installed GRLIR, by calling:
  * `python -m pip install -U vsgrlir`
  * `python -m vsgrlir`
* installed AnimeSR by calling:
  * `python -m pip install -U vsanimesr`
* installed 'vsrealesrgan':
  * `python -m pip install -U vsrealesrgan`
  * `python -m vsrealesrgan`
* installed propainter 
  * `python -m pip install https://github.com/dan64/vs-propainter/releases/download/v1.2.0/vspropainter-1.2.0-py3-none-any.whl`
  * copied 'propainter-weights' into 'vspropainter\weights'-folder
* installed CodeFormer by calling:
  * `python -m pip install dlib-19.24.99-cp312-cp312-win_amd64.whl` (from https://forum.selur.net/attachment.php?aid=2522)
  * `python -m pip install vscodeformer-1.0.2-py3-none-any.whl` (from https://forum.selur.net/attachment.php?aid=2552)
  * `python -m vscodeformer` 
* installed vsddcolor
  * `python -m pip install -U vsddcolor`
  * `python -m vsddcolor` 
* installed BasicVSR++:
  * `python -m pip install aliyun-python-sdk-core-v3==2.13.10`
  * `python -m pip install -U openmim`
  *  extract https://forum.selur.net/attachment.php?aid=2531 into Lib\site-packages folder
  * `python -m pip install -U vsbasicvsrpp`
  * `python -m vsbasicvsrpp`
* installed deep deinterlace:
  * copied vs_deepdeinterlace from the project into Lib\site-packages-folder
  * `python -m pip install positional_encodings`
* installed vsddeoldify
  * `python -m pip install https://github.com/dan64/vs-deoldify/releases/download/v4.0.1/vsdeoldify-4.0.1-py3-none-any.whl`
  * `python -m pip install scikit-image`
  * `python -m pip install numba`
  * `python -m pip install opencv-contrib-python`
  * extracted  vsdeoldify-4.5.0_RC16.zip into Lib\site-packages-folder
  * copied 'deoldiy models' to models-folder  (ColorizeArtistic_gen.pth, ColorizeStable_gen.pth, ColorizeVideo_gen.pth)
  * copied colorization_checkpoints into deoldify/deepex-folder
  * extracted spatial_correlation_sampler-0.5.0-py312-cp312-win_amd64.whl.zip into "Lib\site-packages"  (from https://forum.selur.net/attachment.php?aid=2718)
  * copied DINOv2FeatureV6_LocalAtten_s2_154000.pth (from  https://github.com/yyang181/colormnet/releases/download/v0.1/DINOv2FeatureV6_LocalAtten_s2_154000.pth) into vsdeoldify\colormnet\weights

* left out gsmfss since it breaks stuff:
  * `python -m pip install -U vsgmfss-fortuna`

=> Sadly, that does not help.
Since Hybrids last version is still on R68, it really is time to update.
Are the steps for vs-deoldify correct, or did I miss something.

Cu Selur