Deoldify Vapoursynth filter

[Selur]

Will do some testing, assuming I get it working tomorrow.
Does it even make sense to use a max_memory_frames values which spans over multiple gops?

[Dan64]

yes it is necessary, in case of GPU out of memory, lowering the max_memory_frames is the only way to fix the problem.

Dan

[Selur]

Okay, but does it make sense for encode_mode 2 ?
Does ist make a difference if I use gop size vs. 10000?

Cu Selur

[Selur]

Updated the deoldify test version.
I took the latest RC and loaded it into my new deoldify test version of Hybrid together with "sample2\thing1.mp4" and it didn't crash.
But with the default:

from vsdeoldify import HAVC_main
clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=0)

I get the first frame colored, and the second frame not.
Frame 0:

Frame 1:

starting with Frame 25

Frame 26:

some color is added.
seems like every 24 frames I get a bit more color.
Frame 72:

=> this seems wrong
Switching to async, coloring isn't good, but there.

Calling:

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)

I get:

Failed to evaluate the script:
Python exception: _upsample_bilinear2d_aa() received an invalid combination of arguments - got (Tensor, list, bool, NoneType), but expected one of:
* (Tensor input, tuple of ints output_size, bool align_corners, tuple of floats scale_factors)
didn't match because some of the arguments have invalid types: (Tensor, !list of [bool, int]!, bool, !NoneType!)
* (Tensor input, tuple of ints output_size, bool align_corners, float scales_h = None, float scales_w = None, *, Tensor out = None)


Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3365, in vapoursynth._vpy_evaluate
File "src\\cython\\vapoursynth.pyx", line 3366, in vapoursynth._vpy_evaluate
File "J:\tmp\tempPreviewVapoursynthFile09_23_22_212.vpy", line 43, in
clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 280, in HAVC_main
clip_colored = HAVC_deepex(clip=clip, clip_ref=clip_ref, method=DeepExMethod, render_speed=DeepExPreset,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 554, in HAVC_deepex
clip_colored = vs_colormnet(clip, clip_ref, clip_sc, image_size=-1, enable_resize=enable_resize,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\vsslib\vsmodels.py", line 61, in vs_colormnet
return vs_colormnet_sync(clip, clip_ref, image_size, enable_resize, frame_propagate, max_memory_frames)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 209, in vs_colormnet_sync
clip_i = vs_colormnet_sync_task(t_start, t_end, vid_render, clip, clip_ref, clip_sc, frame_propagate, ref_weight)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 177, in vs_colormnet_sync_task
out_img = vid_render.colorize_frame(i, img_orig)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 190, in colorize_frame
data = self.get_image(ti, frame_i, self.ref_img)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 259, in get_image
img = self.im_transform(img)
^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torchvision\transforms\transforms.py", line 95, in __call__
img = t(img)
^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torchvision\transforms\transforms.py", line 354, in forward
return F.resize(img, self.size, self.interpolation, self.max_size, self.antialias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torchvision\transforms\functional.py", line 479, in resize
return F_t.resize(img, size=output_size, interpolation=interpolation.value, antialias=antialias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torchvision\transforms\_functional_tensor.py", line 465, in resize
img = interpolate(img, size=size, mode=interpolation, align_corners=align_corners, antialias=antialias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\functional.py", line 4565, in interpolate
return torch._C._nn._upsample_bilinear2d_aa(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: _upsample_bilinear2d_aa() received an invalid combination of arguments - got (Tensor, list, bool, NoneType), but expected one of:
* (Tensor input, tuple of ints output_size, bool align_corners, tuple of floats scale_factors)
didn't match because some of the arguments have invalid types: (Tensor, !list of [bool, int]!, bool, !NoneType!)
* (Tensor input, tuple of ints output_size, bool align_corners, float scales_h = None, float scales_w = None, *, Tensor out = None)

that seems like a bug.

Cu Selur

[Selur]

Using:

clip = HAVC_main(clip=clip, Preset="medium", EnableDeepEx=True, DeepExRefMerge=2, ScFrameDir="J:/tmp", ScMinFreq=1, DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=12)

the first 12 frames are colored, when trying to access frame 12, I get:

2024-09-21 09:53:06.423
Error on frame 12 request:

Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3194, in vapoursynth.publicFunction
File "src\\cython\\vapoursynth.pyx", line 3196, in vapoursynth.publicFunction
File "src\\cython\\vapoursynth.pyx", line 829, in vapoursynth.FuncData.__call__
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 319, in clip_colorizer
img_colored = colorizer.get_img_from_cache(n, clip_v, clip_ref, clip_sc, weight)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 134, in get_img_from_cache
output = self.colorizer.colorize_batch_frames(frame_list=img_list, ref_list=ref_list,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 175, in colorize_batch_frames
col_i = img_weighted_merge(col_i, ref_i, weight)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_utils.py", line 54, in img_weighted_merge
img_m = np.multiply(img1_np, 1 - weight).clip(0, 255).astype(int) + np.multiply(img2_np, weight).clip(0,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'

=> seems like a bug.

Cu Selur

[Selur]

Side question:
Is installing

python -m pip install scikit-image
python -m pip install numba
python -m pip install opencv-contrib-python

still necessary for vsdeoldify?
I got it in my notes, but https://github.com/dan64/vs-deoldify/tree/main doesn't mention it.

Cu Selur

[Dan64]

Is installing

python -m pip install scikit-image
python -m pip install numba
python -m pip install opencv-contrib-python

still necessary for vsdeoldify?
I got it in my notes, but https://github.com/dan64/vs-deoldify/tree/main doesn't mention it.

I think so, nothing is changed regarding Deep-Exemplar and these packages were specified in my post 468

These packages are included in the file requirements.txt

Also these packages were not mentioned in https://github.com/zhangmozhe/Deep-Exemp...lorization

So please keep them,

---

Using:

clip = HAVC_main(clip=clip, Preset="medium", EnableDeepEx=True, DeepExRefMerge=2, ScFrameDir="J:/tmp", ScMinFreq=1, DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=12)

=> seems like a bug.

I had the bad idea to release some untested last minute changes.

The attached RC10, should fix all the observed problems.

Dan

[Selur]

Using the default when EnableDeepEx is enabled:

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=0)

I still have issues with the coloring. Seems like chroma any luma plane are out of sync.


























Cu Selur

[Selur]

Using

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=12)

Now does not crash, but also has some sort of async, which seems to reset after DeepExMaxMemFrames.



























Cu Selur

[Dan64]

It is strange, I don't observe this problem.
Try the attached RC11.
If the problem still persist, please provide your test sample that I will check it.

Dan.

P.S.
To enable the use of Ref Frames, you have to provide in input "DeepExMethod=1"  -> "HAVC + RF same as video"