Deoldify Vapoursynth filter - Printable Version

Deoldify Vapoursynth filter - Printable Version

+- Selur's Little Message Board (https://forum.selur.net)
+-- Forum: Talk, Talk, Talk (https://forum.selur.net/forum-5.html)
+--- Forum: Small Talk (https://forum.selur.net/forum-7.html)
+--- Thread: Deoldify Vapoursynth filter (/thread-3595.html)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143

RE: Deoldify Vapoursynth filter - Selur - 20.09.2024

Will do some testing, assuming I get it working tomorrow.
Does it even make sense to use a max_memory_frames values which spans over multiple gops?

RE: Deoldify Vapoursynth filter - Dan64 - 20.09.2024

yes it is necessary, in case of GPU out of memory, lowering the max_memory_frames is the only way to fix the problem.

Dan

RE: Deoldify Vapoursynth filter - Selur - 21.09.2024

Okay, but does it make sense for encode_mode 2 ?
Does ist make a difference if I use gop size vs. 10000?

Cu Selur

RE: Deoldify Vapoursynth filter - Selur - 21.09.2024

Updated the deoldify test version.
I took the latest RC and loaded it into my new deoldify test version of Hybrid together with "sample2\thing1.mp4" and it didn't crash. Smile

But with the default:

from vsdeoldify import HAVC_main
clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=0)

I get the first frame colored, and the second frame not.
Frame 0:

Frame 1:

starting with Frame 25

Frame 26:

some color is added.
seems like every 24 frames I get a bit more color.
Frame 72:

=> this seems wrong Smile

Switching to async, coloring isn't good, but there.

Calling:

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)

I get:

Failed to evaluate the script:
Python exception: _upsample_bilinear2d_aa() received an invalid combination of arguments - got (Tensor, list, bool, NoneType), but expected one of:
* (Tensor input, tuple of ints output_size, bool align_corners, tuple of floats scale_factors)
didn't match because some of the arguments have invalid types: (Tensor, !list of [bool, int]!, bool, !NoneType!)
* (Tensor input, tuple of ints output_size, bool align_corners, float scales_h = None, float scales_w = None, *, Tensor out = None)


Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3365, in vapoursynth._vpy_evaluate
File "src\\cython\\vapoursynth.pyx", line 3366, in vapoursynth._vpy_evaluate
File "J:\tmp\tempPreviewVapoursynthFile09_23_22_212.vpy", line 43, in 
clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=2, DeepExMaxMemFrames=0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 280, in HAVC_main
clip_colored = HAVC_deepex(clip=clip, clip_ref=clip_ref, method=DeepExMethod, render_speed=DeepExPreset,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 554, in HAVC_deepex
clip_colored = vs_colormnet(clip, clip_ref, clip_sc, image_size=-1, enable_resize=enable_resize,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\vsslib\vsmodels.py", line 61, in vs_colormnet
return vs_colormnet_sync(clip, clip_ref, image_size, enable_resize, frame_propagate, max_memory_frames)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 209, in vs_colormnet_sync
clip_i = vs_colormnet_sync_task(t_start, t_end, vid_render, clip, clip_ref, clip_sc, frame_propagate, ref_weight)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 177, in vs_colormnet_sync_task
out_img = vid_render.colorize_frame(i, img_orig)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 190, in colorize_frame
data = self.get_image(ti, frame_i, self.ref_img)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 259, in get_image
img = self.im_transform(img)
^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torchvision\transforms\transforms.py", line 95, in __call__
img = t(img)
^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1716, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1727, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torchvision\transforms\transforms.py", line 354, in forward
return F.resize(img, self.size, self.interpolation, self.max_size, self.antialias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torchvision\transforms\functional.py", line 479, in resize
return F_t.resize(img, size=output_size, interpolation=interpolation.value, antialias=antialias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torchvision\transforms\_functional_tensor.py", line 465, in resize
img = interpolate(img, size=size, mode=interpolation, align_corners=align_corners, antialias=antialias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\functional.py", line 4565, in interpolate
return torch._C._nn._upsample_bilinear2d_aa(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: _upsample_bilinear2d_aa() received an invalid combination of arguments - got (Tensor, list, bool, NoneType), but expected one of:
* (Tensor input, tuple of ints output_size, bool align_corners, tuple of floats scale_factors)
didn't match because some of the arguments have invalid types: (Tensor, !list of [bool, int]!, bool, !NoneType!)
* (Tensor input, tuple of ints output_size, bool align_corners, float scales_h = None, float scales_w = None, *, Tensor out = None)

that seems like a bug.

Cu Selur

RE: Deoldify Vapoursynth filter - Selur - 21.09.2024

Using:

clip = HAVC_main(clip=clip, Preset="medium", EnableDeepEx=True, DeepExRefMerge=2, ScFrameDir="J:/tmp", ScMinFreq=1, DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=12)

the first 12 frames are colored, when trying to access frame 12, I get:

2024-09-21 09:53:06.423
Error on frame 12 request:

Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3194, in vapoursynth.publicFunction
File "src\\cython\\vapoursynth.pyx", line 3196, in vapoursynth.publicFunction
File "src\\cython\\vapoursynth.pyx", line 829, in vapoursynth.FuncData.__call__
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 319, in clip_colorizer
img_colored = colorizer.get_img_from_cache(n, clip_v, clip_ref, clip_sc, weight)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\__init__.py", line 134, in get_img_from_cache
output = self.colorizer.colorize_batch_frames(frame_list=img_list, ref_list=ref_list,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_render.py", line 175, in colorize_batch_frames
col_i = img_weighted_merge(col_i, ref_i, weight)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\colormnet\colormnet_utils.py", line 54, in img_weighted_merge
img_m = np.multiply(img1_np, 1 - weight).clip(0, 255).astype(int) + np.multiply(img2_np, weight).clip(0,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'

=> seems like a bug.

Cu Selur

RE: Deoldify Vapoursynth filter - Selur - 21.09.2024

Side question:
Is installing

python -m pip install scikit-image
python -m pip install numba
python -m pip install opencv-contrib-python

still necessary for vsdeoldify?
I got it in my notes, but https://github.com/dan64/vs-deoldify/tree/main doesn't mention it.

Cu Selur

RE: Deoldify Vapoursynth filter - Dan64 - 21.09.2024

(21.09.2024, 18:43)Selur Wrote: Is installing

python -m pip install scikit-image python -m pip install numba python -m pip install opencv-contrib-python
still necessary for vsdeoldify?
I got it in my notes, but https://github.com/dan64/vs-deoldify/tree/main doesn't mention it.

I think so, nothing is changed regarding Deep-Exemplar and these packages were specified in my post 468

These packages are included in the file requirements.txt

Also these packages were not mentioned in https://github.com/zhangmozhe/Deep-Exemplar-based-Video-Colorization

So please keep them,

(21.09.2024, 09:53)Selur Wrote: Using:

clip = HAVC_main(clip=clip, Preset="medium", EnableDeepEx=True, DeepExRefMerge=2, ScFrameDir="J:/tmp", ScMinFreq=1, DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=12)
=> seems like a bug.

I had the bad idea to release some untested last minute changes.

The attached RC10, should fix all the observed problems.

Dan

RE: Deoldify Vapoursynth filter - Selur - 22.09.2024

Using the default when EnableDeepEx is enabled:

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=0)

I still have issues with the coloring. Seems like chroma any luma plane are out of sync.

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-0.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-1.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-2.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-3.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-4.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-5.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-6.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-7.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-8.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-9.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-10.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-11.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-12.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-13.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-14.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-15.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-16.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-17.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-18.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-19.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-20.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-21.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-23.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-24.png]

[Image: temp-Preview-Vapoursynth-File07-12-35-309-vpy-25.png]

Cu Selur

RE: Deoldify Vapoursynth filter - Selur - 22.09.2024

Using

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=12)

Now does not crash, but also has some sort of async, which seems to reset after DeepExMaxMemFrames.

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-1.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-2.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-3.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-4.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-12.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-13.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-14.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-15.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-16.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-17.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-18.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-19.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-20.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-21.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-22.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-23.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-24.png]

[Image: temp-Preview-Vapoursynth-File07-22-40-786-vpy-25.png]

Cu Selur

RE: Deoldify Vapoursynth filter - Dan64 - 22.09.2024

It is strange, I don't observe this problem.
Try the attached RC11.
If the problem still persist, please provide your test sample that I will check it.

Dan.

P.S.
To enable the use of Ref Frames, you have to provide in input "DeepExMethod=1" -> "HAVC + RF same as video"