Deoldify Vapoursynth filter - Printable Version

Deoldify Vapoursynth filter - Printable Version

+- Selur's Little Message Board (https://forum.selur.net)
+-- Forum: Talk, Talk, Talk (https://forum.selur.net/forum-5.html)
+--- Forum: Small Talk (https://forum.selur.net/forum-7.html)
+--- Thread: Deoldify Vapoursynth filter (/thread-3595.html)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81

RE: Deoldify Vapoursynth filter - Selur - 20.09.2024

Will do some testing, assuming I get it working tomorrow.
Does it even make sense to use a max_memory_frames values which spans over multiple gops?

RE: Deoldify Vapoursynth filter - Dan64 - 20.09.2024

yes it is necessary, in case of GPU out of memory, lowering the max_memory_frames is the only way to fix the problem.

Dan

RE: Deoldify Vapoursynth filter - Selur - 21.09.2024

Okay, but does it make sense for encode_mode 2 ?
Does ist make a difference if I use gop size vs. 10000?

Cu Selur

RE: Deoldify Vapoursynth filter - Selur - 21.09.2024

Updated the deoldify test version.
I took the latest RC and loaded it into my new deoldify test version of Hybrid together with "sample2\thing1.mp4" and it didn't crash. Smile

But with the default:

from vsdeoldify import HAVC_main

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=0)

I get the first frame colored, and the second frame not.
Frame 0:

Frame 1:

starting with Frame 25

Frame 26:

some color is added.
seems like every 24 frames I get a bit more color.
Frame 72:

=> this seems wrong Smile

Switching to async, coloring isn't good, but there.

Calling:

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)

I get:

Failed to evaluate the script:

Python exception: _upsample_bilinear2d_aa() received an invalid combination of arguments - got (Tensor, list, bool, NoneType), but expected one of:

* (Tensor input, tuple of ints output_size, bool align_corners, tuple of floats scale_factors)

didn't match because some of the arguments have invalid types: (Tensor, !list of [bool, int]!, bool, !NoneType!)

* (Tensor input, tuple of ints output_size, bool align_corners, float scales_h = None, float scales_w = None, *, Tensor out = None)

Traceback (most recent call last):

File "src\\cython\\vapoursynth.pyx", line 3365, in vapoursynth._vpy_evaluate

File "src\\cython\\vapoursynth.pyx", line 3366, in vapoursynth._vpy_evaluate

File "J:\tmp\tempPreviewVapoursynthFile09_23_22_212.vpy", line 43, in 

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 280, in HAVC_main

clip_colored = HAVC_deepex(clip=clip, clip_ref=clip_ref, method=DeepExMethod, render_speed=DeepExPreset,

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 554, in HAVC_deepex

clip_colored = vs_colormnet(clip, clip_ref, clip_sc, image_size=-1, enable_resize=enable_resize,

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\vsslib\vsmodels.py", line 61, in vs_colormnet

return vs_colormnet_sync(clip, clip_ref, image_size, enable_resize, frame_propagate, max_memory_frames)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 209, in vs_colormnet_sync

clip_i = vs_colormnet_sync_task(t_start, t_end, vid_render, clip, clip_ref, clip_sc, frame_propagate, ref_weight)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 177, in vs_colormnet_sync_task

out_img = vid_render.colorize_frame(i, img_orig)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 190, in colorize_frame

data = self.get_image(ti, frame_i, self.ref_img)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 259, in get_image

img = self.im_transform(img)

^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torchvision\transforms\transforms.py", line 95, in __call__

img = t(img)

^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl

return self._call_impl(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl

return forward_call(*args, **kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torchvision\transforms\transforms.py", line 354, in forward

return F.resize(img, self.size, self.interpolation, self.max_size, self.antialias)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torchvision\transforms\functional.py", line 479, in resize

return F_t.resize(img, size=output_size, interpolation=interpolation.value, antialias=antialias)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torchvision\transforms\_functional_tensor.py", line 465, in resize

img = interpolate(img, size=size, mode=interpolation, align_corners=align_corners, antialias=antialias)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\functional.py", line 4565, in interpolate

return torch._C._nn._upsample_bilinear2d_aa(

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

TypeError: _upsample_bilinear2d_aa() received an invalid combination of arguments - got (Tensor, list, bool, NoneType), but expected one of:

* (Tensor input, tuple of ints output_size, bool align_corners, tuple of floats scale_factors)

didn't match because some of the arguments have invalid types: (Tensor, !list of [bool, int]!, bool, !NoneType!)

* (Tensor input, tuple of ints output_size, bool align_corners, float scales_h = None, float scales_w = None, *, Tensor out = None)

that seems like a bug.

Cu Selur

RE: Deoldify Vapoursynth filter - Selur - 21.09.2024

Using:

clip = HAVC_main(clip=clip, Preset="medium", EnableDeepEx=True, DeepExRefMerge=2, ScFrameDir="J:/tmp", ScMinFreq=1, DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=12)

the first 12 frames are colored, when trying to access frame 12, I get:

2024-09-21 09:53:06.423

Error on frame 12 request:

Traceback (most recent call last):

File "src\\cython\\vapoursynth.pyx", line 3194, in vapoursynth.publicFunction

File "src\\cython\\vapoursynth.pyx", line 3196, in vapoursynth.publicFunction

File "src\\cython\\vapoursynth.pyx", line 829, in vapoursynth.FuncData.__call__

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 319, in clip_colorizer

img_colored = colorizer.get_img_from_cache(n, clip_v, clip_ref, clip_sc, weight)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 134, in get_img_from_cache

output = self.colorizer.colorize_batch_frames(frame_list=img_list, ref_list=ref_list,

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 175, in colorize_batch_frames

col_i = img_weighted_merge(col_i, ref_i, weight)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_utils.py", line 54, in img_weighted_merge

img_m = np.multiply(img1_np, 1 - weight).clip(0, 255).astype(int) + np.multiply(img2_np, weight).clip(0,

^^^^^^^^^^^^^^^^^^^^^^^^^^^^

TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'

=> seems like a bug.

Cu Selur

RE: Deoldify Vapoursynth filter - Selur - 21.09.2024

Side question:
Is installing

python -m pip install scikit-image

python -m pip install numba

python -m pip install opencv-contrib-python

still necessary for vsdeoldify?
I got it in my notes, but https://github.com/dan64/vs-deoldify/tree/main doesn't mention it.

Cu Selur

RE: Deoldify Vapoursynth filter - Dan64 - 21.09.2024

(21.09.2024, 18:43)Selur Wrote: Is installing

python -m pip install scikit-image python -m pip install numba python -m pip install opencv-contrib-python
still necessary for vsdeoldify?
I got it in my notes, but https://github.com/dan64/vs-deoldify/tree/main doesn't mention it.

I think so, nothing is changed regarding Deep-Exemplar and these packages were specified in my post 468

These packages are included in the file requirements.txt

Also these packages were not mentioned in https://github.com/zhangmozhe/Deep-Exemplar-based-Video-Colorization

So please keep them,

(21.09.2024, 09:53)Selur Wrote: Using:

clip = HAVC_main(clip=clip, Preset="medium", EnableDeepEx=True, DeepExRefMerge=2, ScFrameDir="J:/tmp", ScMinFreq=1, DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=12)
=> seems like a bug.

I had the bad idea to release some untested last minute changes.

The attached RC10, should fix all the observed problems.

Dan

RE: Deoldify Vapoursynth filter - Selur - 22.09.2024

Using the default when EnableDeepEx is enabled:

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=0)

I still have issues with the coloring. Seems like chroma any luma plane are out of sync.

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-0.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-1.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-2.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-3.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-4.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-5.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-6.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-7.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-8.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-9.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-10.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-11.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-12.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-13.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-14.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-15.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-16.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-17.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-18.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-19.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-20.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-21.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-23.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-24.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-25.png]

Cu Selur

RE: Deoldify Vapoursynth filter - Selur - 22.09.2024

Using

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=12)

Now does not crash, but also has some sort of async, which seems to reset after DeepExMaxMemFrames.

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-1.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-2.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-3.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-4.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-12.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-13.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-14.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-15.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-16.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-17.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-18.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-19.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-20.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-21.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-22.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-23.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-24.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-25.png]

Cu Selur

RE: Deoldify Vapoursynth filter - Dan64 - 22.09.2024

It is strange, I don't observe this problem.
Try the attached RC11.
If the problem still persist, please provide your test sample that I will check it.

Dan.

P.S.
To enable the use of Ref Frames, you have to provide in input "DeepExMethod=1" -> "HAVC + RF same as video"