Deoldify Vapoursynth filter

[Selur]

In Hybrid I differentiate between:
a. the filter defaults, this allow to know which settings are the defaults in the filter and can be left auf.
b. the settings Hybrid uses by defaults
I'll adjust my dev version to "dark_darkness=[True,0.4,0.8,0.8,0.8]" for the settings uses by default.
Will adjust the filter defaults in Hybrid, when vs-deoldify changes.

Cu Selur

[zspeciman]

Hello, I have the following error in the rev 2024.03.18.1 / VapoursynthR65_torch_2024.03.17

DDcolor by itself works fine when it is enabled

The Moment I enable DeOldify / DeOdify Only (DDcolor not checked) I get this

The Debug output just has this:
2024.03.18 - 08:58:26_Windows 10 Version 1809 (64bit)_2024.03.18.1 - level 9: Output is mp4box compatible,.. -> changing current container to mp4,..
2024.03.18 - 08:58:26_Windows 10 Version 1809 (64bit)_2024.03.18.1 - level 9: outputDir set to(2): C:\Program Files\Hybrid

The preview mode has more info:
2024-03-18 09:06:56.586
Failed to evaluate the script:
Python exception: No attribute with the name misc exists. Did you mistype a plugin namespace?

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 3115, in vapoursynth._vpy_evaluate
File "src\cython\vapoursynth.pyx", line 3116, in vapoursynth._vpy_evaluate
File "C:\Users\Yoyo-Desktop\AppData\Local\Temp\tempPreviewVapoursynthFile09_06_43_077.vpy", line 29, in
clip = ddeoldify(clip=clip, model=0, sat=[1.00,1], hue=[0.00,0], chroma_resize=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py", line 260, in ddeoldify
clip_colored = vs_clip_color_stabilizer(clip_colored, nframes = colstab_nframes, mode=colstab_mode, scenechange = colstab_scenechange)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\deoldify\adjust.py", line 905, in vs_clip_color_stabilizer
clip_yuv = vs.core.misc.AverageFrames(clip_yuv, weight_list, scenechange = True, planes=[1,2])
^^^^^^^^^^^^
File "src\cython\vapoursynth.pyx", line 2628, in vapoursynth._CoreProxy.__getattr__
File "src\cython\vapoursynth.pyx", line 2483, in vapoursynth.Core.__getattr__
AttributeError: No attribute with the name misc exists. Did you mistype a plugin namespace?



I tried different situations.  DeOldify and DDcolor both checked, and different DDColor Combine ones, they all have errors.
In the previous version, Hybrid_2024.03.03.1.  DDcolor and DeOldify each worked fine by themselves.

[Dan64]

In the current version there is a bug.
As work around, it is enough that you enable "Chroma Stabilization"

Dan

[Selur]

For explaination:
Hybrid only loads MiscFilters when Color Stabilization is enabled, problem is atm. Hybrid also doesn't add the 'color_stabilizer: list' unless Color Stabilization is enabled (this is the bug). This causes vsDeoldify to assume the default 'color_stabilizer: list = [True, False, False, 11, 'center', True]' and thus trying to use Misc.

[Selur]

Looking at the GPU usage while using DeOldify:

# adding colors using DeOldify
from vsdeoldify import ddeoldify
clip = ddeoldify(clip=clip, model=0, sat=[1.00,1], hue=[0.00,0], chroma_resize=True, color_stabilizer=[False,False,False,11,"center",True])

I noticed that the usage is rather 'spikey'.

Since the processing is frame-by-frame without taking the neighbors into account I tried the following:

# adding colors using DeOldify
from vsdeoldify import ddeoldify
# split frames and stack them
even = clip[::2] # even frames
odd =  clip[1::2] # odd frames
clip = core.std.StackHorizontal([even, odd]) # stack even and odd

clip = ddeoldify(clip=clip, model=0, sat=[1.00,1], hue=[0.00,0], chroma_resize=True, color_stabilizer=[False,False,False,11,"center",True]) # do the processing

# split frames and interleave them again
even = core.std.Crop(clip, left=0, right = clip.width/2)  
odd = core.std.Crop(clip, left=clip.width/2, right = 0)
clip = core.std.Interleave([even,odd])

this does seem for me to:
a. speed things up quite a bit
b. smooth the gpu usage
(I only tested this in vsViewer not for acctual encoding)

So general idea is to process two frames at a time by stacking them horizontally and split them after processing.

It might be worth some further testing whether
a. this effect can be produced by others
b. it might be worth adding this as an option to vs-deoldify

=> Can anyone reproduce the effect?
(I only expect this to work for SD content. I expect HD and higher resolutions to be too taxing for this to work.)

Cu Selur

[Dan64]

I seems a good Idea. As you know with render_factor=24 and chroma_resize=true all the filters operations, including the colorization process are performed on frame 384x384.
Could you provide the GPU usage with the staking of 2 frames ?
I also written to Jantic to know if there is any plan to use 16bit tensors for inference.

Dan

[Dan64]

I don't know how Vapoursynth can use the future frames if they has not been colored yet.

in your code vs_clip_color_stabilizer is always processed after the colorization

It seems that this feature is not working very well, I noted that if nframes > 7, the averaged image starts to becomes gray, as  if it was losing the chroma component.

Dan

P.S.
I suggest to don't use more than 7 frames in the chroma stabilization filter.

[Selur]

using:

from vsdeoldify import ddeoldify
even = clip[::2]
odd =  clip[1::2]
clip = core.std.StackHorizontal([even, odd])
clip = ddeoldify(clip=clip, model=0, sat=[1.00,1], hue=[0.00,0], chroma_resize=True, color_stabilizer=[False,False,False,11,"center",True])
even = core.std.Crop(clip, left=0, right = clip.width/2)
odd = core.std.Crop(clip, left=clip.width/2, right = 0)
clip = core.std.Interleave([even,odd])

and letting it loop:

usage >= 90%
just using:

clip = ddeoldify(clip=clip, model=0, sat=[1.00,1], hue=[0.00,0], chroma_resize=True, color_stabilizer=[False,False,False,11,"center",True])

Not sure whether it's a vsViewer problem, I used:

VSPipe.exe --progress -c y4m c:\Users\Selur\Desktop\test.vpy NUL

normal (= just deoldify):

Script evaluation done in 11.18 seconds
Output 429 frames in 42.74 seconds (10.04 fps)

stacked (= split, stack, deoldify, split, interleave):

Script evaluation done in 8.62 seconds
Output 430 frames in 22.62 seconds (19.01 fps)

So this is not a problem with vsViewer.

Cu Selur

[Dan64]

Speed improvement ?

[Selur]

See the vspipe output. 10 vs 19fps