Deoldify Vapoursynth filter

[Dan64]

I have the problem.

Steps to reproduce it.

1) Import the attached image
2) Color->Tewak->Saturation=0
3) Preview

The generated script is

# Imports import vapoursynth as vs # getting Vapoursynth core import sys import os core = vs.core # Import scripts folder scriptPath = 'D:/Programs/Hybrid/64bit/vsscripts' sys.path.insert(0, os.path.abspath(scriptPath)) # Loading Plugins core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/Support/fmtconv.dll") core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/Support/libimwri.dll") # Import scripts import adjust # source: 'D:/PProjects/vs-deoldify_dev/TestCOL1.jpg' # current color space: YUV420P8, bit depth: 8, resolution: 1280x1490, fps: 25, scanorder: progressive, yuv luminance scale: full, matrix: 709 # Loading D:\PProjects\vs-deoldify_dev\TestCOL1.jpg using vsImageReader clip = core.imwri.Read(["D:/PProjects/vs-deoldify_dev/TestCOL1.jpg"]) clip = core.std.Loop(clip=clip, times=100) frame = clip.get_frame(0) # Setting detected color matrix (709). clip = core.std.SetFrameProps(clip, _Matrix=1) # Setting color transfer (2020), if it is not set. if '_Transfer' not in frame.props or not frame.props['_Transfer']:   clip = core.std.SetFrameProps(clip, _Transfer=15) # Setting color primaries info (to 2020), if it is not set. if '_Primaries' not in frame.props or not frame.props['_Primaries']:   clip = core.std.SetFrameProps(clip, _Primaries=9) # Setting color range to PC (full) range. clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0) # making sure frame rate is set to 25 clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1) clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive # Color Adjustment clip = adjust.Tweak(clip=clip, hue=0.00, sat=0.00, cont=1.00, coring=True) # Resizing using fmtconv - spline36 clip = core.fmtc.resample(clip=clip, kernel="spline36", w=1280, h=1488, interlaced=False, interlacedd=False) # resolution 1280x1488 before YUV420P8 after YUV420P16 # adjusting output color from: YUV420P16 to YUV420P10 for x265Model clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="full", dither_type="error_diffusion") # set output frame rate to 25fps (progressive) clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1) # Output clip.set_output()

And in preview I get
 

2024-03-03 14:41:44.061 Failed to evaluate the script: Python exception: Tweak: RGB clips are not accepted. Traceback (most recent call last): File "src\cython\vapoursynth.pyx", line 3115, in vapoursynth._vpy_evaluate File "src\cython\vapoursynth.pyx", line 3116, in vapoursynth._vpy_evaluate File "D:\PProjects\vs-deoldify_dev\tempPreviewVapoursynthFile14_41_43_544.vpy", line 35, in clip = adjust.Tweak(clip=clip, hue=0.00, sat=0.00, cont=1.00, coring=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Programs\Hybrid\64bit\vsscripts\adjust.py", line 13, in Tweak raise vs.Error("Tweak: RGB clips are not accepted.") vapoursynth.Error: Tweak: RGB clips are not accepted.

Dan

[Selur]

Problem is that, MediaInfo reports the image as:

Image Count : 166 Count of stream of this kind : 1 Kind of stream : Image Kind of stream : Image Stream identifier : 0 Format : JPEG Format : JPEG Commercial name : JPEG Internet media type : image/jpeg Width : 1280 Width : 1 280 pixels Height : 1490 Height : 1 490 pixels Color space : YUV Chroma subsampling : 4:2:0 Bit depth : 8 Bit depth : 8 bits Compression mode : Lossy Compression mode : Lossy Stream size : 442245 Stream size : 432 KiB (100%) Stream size : 432 KiB Stream size : 432 KiB Stream size : 432 KiB Stream size : 431.9 KiB Stream size : 432 KiB (100%) Proportion of this stream : 1.00000

but,

clip = core.imwri.Read(["C:/Users/Selur/Desktop/TestCOL1.jpg"])

imports the image as RGB
-> I'll fix it.

[Dan64]

I released a new version: https://github.com/dan64/vs-deoldify/rel...tag/v1.1.3

I added conversion of clip format to RGB24 when necessary. 

In this way you can revert the changes applied in Hybrid to handle the special case GRAY8.
I can imagine that managing the conversion for all the filters in Hybrid is very complicated.
My approach to problems is to not complicate things unless absolutely necessary.

Thanks,
Dan

[Selur]

No need, to undo the changes.
Send you a new link.

Cu Selur

[Selur]

if clip.format.id != vs.RGB24: # clip not in RGB24 format, it will be converted clip = clip.resize.Bicubic(format=vs.RGB24, range_s="limited")

did you test this with different inputs?
iirc. if you do YUV to RGB conversions (or the other way around), one needs to specify the input matrix.
(maybe they changed this, that as a fallback, the frame properties are checked,...)

Cu Selur

[Dan64]

Not tested in all situations, also I added the control. 

The original deoldify used a different approach, the images were stored in the filesystems using ffmpeg, this implies that to convert a 100min films were required 20gb to store both the B&W and Color frames.

I introduced the image colorize directly in the memory, without the need to use the filesystem as temporary storage.
I also had to write the conversion from PIL image to frame and vice-versa.

Dan

[Selur]

=> I would recommend sticking with only accepting RGB24 input.
btw. does DeOldify look at multiple images or only at one at a time? (the 'Video' model makes me hope that it looks at image sequences and not one at a time)

Cu Selur

[zspeciman]

That is impressive work Dan, how much faster is your approach when it comes to videos; memory vs filesystem?
I hope Selur will be able to integrate it into Hybrid, that will be huge

[Selur]

File vs memory shouldn't really matter in regard of speed, since the slow part is probably not the accessing of the images, but the processing of the images.

Cu Selur

[Dan64]

=> I would recommend sticking with only accepting RGB24 input.
btw. does DeOldify look at multiple images or only at one at a time? (the 'Video' model makes me hope that it looks at image sequences and not one at a time)

I do agree with you, I added the the conversion only to handle some odd situation not managed by Hybrid, but Hybrid remain my main umbrella for conversion problems.

I mostly rewrite the module "visualize.py" that was delegate to manage the conversion using the filesystem. It was the most difficult part to write the Vapoursynth filter.

The Jupiter version of Deoldify first exports all the B&W frames in a temporary directory, then in a second step it read a frame a time, colorize it and save in another temporary directory. Finally using ffmpeg the colorized frames are converted in a movie.

This approach introduce an overhead which is not necessary using Vapoursynth.    

In the table below I put a comparison between the Jupiter version of Deoldify and Vapoursynth, to encode a 100min of 720p movie.



The storage necessary to store the frames is about 20Gb (10Gb for B&W and 10Gb for colored frames).

The total time spent by Jupiter to encode the movie is about 8h, while the Vapoursynth version is able to encode the movie in about 6h:41m

But now thanks to Hybrid and to this version of Deoldify filter (which I called DDeoldify because DDColor is integrate in it) a can finally start to colorize my preferred B&W movies.  
 

Thanks,
Dan