Deoldify Vapoursynth filter

[Dan64]

I have the problem.

Steps to reproduce it.

1) Import the attached image
2) Color->Tewak->Saturation=0
3) Preview

The generated script is

# Imports
import vapoursynth as vs
# getting Vapoursynth core
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'D:/Programs/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/Support/libimwri.dll")
# Import scripts
import adjust
# source: 'D:/PProjects/vs-deoldify_dev/TestCOL1.jpg'
# current color space: YUV420P8, bit depth: 8, resolution: 1280x1490, fps: 25, scanorder: progressive, yuv luminance scale: full, matrix: 709
# Loading D:\PProjects\vs-deoldify_dev\TestCOL1.jpg using vsImageReader
clip = core.imwri.Read(["D:/PProjects/vs-deoldify_dev/TestCOL1.jpg"])
clip = core.std.Loop(clip=clip, times=100)
frame = clip.get_frame(0)
# Setting detected color matrix (709).
clip = core.std.SetFrameProps(clip, _Matrix=1)
# Setting color transfer (2020), if it is not set.
if '_Transfer' not in frame.props or not frame.props['_Transfer']:
  clip = core.std.SetFrameProps(clip, _Transfer=15)
# Setting color primaries info (to 2020), if it is not set.
if '_Primaries' not in frame.props or not frame.props['_Primaries']:
  clip = core.std.SetFrameProps(clip, _Primaries=9)
# Setting color range to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
# Color Adjustment
clip = adjust.Tweak(clip=clip, hue=0.00, sat=0.00, cont=1.00, coring=True)
# Resizing using fmtconv - spline36
clip = core.fmtc.resample(clip=clip, kernel="spline36", w=1280, h=1488, interlaced=False, interlacedd=False) # resolution 1280x1488 before YUV420P8 after YUV420P16
# adjusting output color from: YUV420P16 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="full", dither_type="error_diffusion")
# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

And in preview I get
 

2024-03-03 14:41:44.061
Failed to evaluate the script:
Python exception: Tweak: RGB clips are not accepted.

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 3115, in vapoursynth._vpy_evaluate
File "src\cython\vapoursynth.pyx", line 3116, in vapoursynth._vpy_evaluate
File "D:\PProjects\vs-deoldify_dev\tempPreviewVapoursynthFile14_41_43_544.vpy", line 35, in
clip = adjust.Tweak(clip=clip, hue=0.00, sat=0.00, cont=1.00, coring=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Programs\Hybrid\64bit\vsscripts\adjust.py", line 13, in Tweak
raise vs.Error("Tweak: RGB clips are not accepted.")
vapoursynth.Error: Tweak: RGB clips are not accepted.

Dan

[Selur]

Problem is that, MediaInfo reports the image as:

Image
Count                                    : 166
Count of stream of this kind             : 1
Kind of stream                           : Image
Kind of stream                           : Image
Stream identifier                        : 0
Format                                   : JPEG
Format                                   : JPEG
Commercial name                          : JPEG
Internet media type                      : image/jpeg
Width                                    : 1280
Width                                    : 1 280 pixels
Height                                   : 1490
Height                                   : 1 490 pixels
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8
Bit depth                                : 8 bits
Compression mode                         : Lossy
Compression mode                         : Lossy
Stream size                              : 442245
Stream size                              : 432 KiB (100%)
Stream size                              : 432 KiB
Stream size                              : 432 KiB
Stream size                              : 432 KiB
Stream size                              : 431.9 KiB
Stream size                              : 432 KiB (100%)
Proportion of this stream                : 1.00000

but,

clip = core.imwri.Read(["C:/Users/Selur/Desktop/TestCOL1.jpg"])

imports the image as RGB
-> I'll fix it.

[Dan64]

I released a new version: https://github.com/dan64/vs-deoldify/rel...tag/v1.1.3

I added conversion of clip format to RGB24 when necessary. 

In this way you can revert the changes applied in Hybrid to handle the special case GRAY8.
I can imagine that managing the conversion for all the filters in Hybrid is very complicated.
My approach to problems is to not complicate things unless absolutely necessary.

Thanks,
Dan

[Selur]

No need, to undo the changes.
Send you a new link.

Cu Selur

[Selur]

if clip.format.id != vs.RGB24:
        # clip not in RGB24 format, it will be converted
        clip = clip.resize.Bicubic(format=vs.RGB24, range_s="limited")

did you test this with different inputs?
iirc. if you do YUV to RGB conversions (or the other way around), one needs to specify the input matrix.
(maybe they changed this, that as a fallback, the frame properties are checked,...)

Cu Selur

[Dan64]

Not tested in all situations, also I added the control. 

The original deoldify used a different approach, the images were stored in the filesystems using ffmpeg, this implies that to convert a 100min films were required 20gb to store both the B&W and Color frames.

I introduced the image colorize directly in the memory, without the need to use the filesystem as temporary storage.
I also had to write the conversion from PIL image to frame and vice-versa.

Dan

[Selur]

=> I would recommend sticking with only accepting RGB24 input.
btw. does DeOldify look at multiple images or only at one at a time? (the 'Video' model makes me hope that it looks at image sequences and not one at a time)

Cu Selur

[zspeciman]

That is impressive work Dan, how much faster is your approach when it comes to videos; memory vs filesystem?
I hope Selur will be able to integrate it into Hybrid, that will be huge

[Selur]

File vs memory shouldn't really matter in regard of speed, since the slow part is probably not the accessing of the images, but the processing of the images.

Cu Selur

[Dan64]

=> I would recommend sticking with only accepting RGB24 input.
btw. does DeOldify look at multiple images or only at one at a time? (the 'Video' model makes me hope that it looks at image sequences and not one at a time)

I do agree with you, I added the the conversion only to handle some odd situation not managed by Hybrid, but Hybrid remain my main umbrella for conversion problems.

I mostly rewrite the module "visualize.py" that was delegate to manage the conversion using the filesystem. It was the most difficult part to write the Vapoursynth filter.

The Jupiter version of Deoldify first exports all the B&W frames in a temporary directory, then in a second step it read a frame a time, colorize it and save in another temporary directory. Finally using ffmpeg the colorized frames are converted in a movie.

This approach introduce an overhead which is not necessary using Vapoursynth.    

In the table below I put a comparison between the Jupiter version of Deoldify and Vapoursynth, to encode a 100min of 720p movie.



The storage necessary to store the frames is about 20Gb (10Gb for B&W and 10Gb for colored frames).

The total time spent by Jupiter to encode the movie is about 8h, while the Vapoursynth version is able to encode the movie in about 6h:41m

But now thanks to Hybrid and to this version of Deoldify filter (which I called DDeoldify because DDColor is integrate in it) a can finally start to colorize my preferred B&W movies.  
 

Thanks,
Dan