Deoldify Vapoursynth filter

[Selur]

I need your support to answer to zspeciman's question.

okay, looking at the scripts.

1st script uses ImageSource, thus frames should already be at RGB24

# adding colors using DeOldify
from vsdeoldify import ddeoldify
clip = ddeoldify(clip=clip, method=0, deoldify_p=[0, 24, 1, 0])

2nd script uses YUV420P8 video source, that is converted to RGB24:

# adjusting color space from YUV420P8 to RGB24 for vsDeOldify
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
# adding colors using DeOldify
from vsdeoldify import ddeoldify
clip = ddeoldify(clip=clip, method=0, deoldify_p=[0, 24, 1, 0])

3rd script uses YUV420P8 video source which is converted to RGB24, but instead of deoldify it uses ddeoldify_main and thus deoldify and ddcolor.

# adjusting color space from YUV420P8 to RGB24 for vsDeOldify
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
# adding colors using DeOldify
from vsdeoldify import ddeoldify_main
clip = ddeoldify_main(clip=clip)

so not really related the first two scripts.

I think that is an issue related to the color space conversion, both my first script and Deoldify original are working on "color space: RGB24, bit depth: 8, yuv luminance scale: full, matrix: 709"
While the second script is using in input "color space: YUV420P8, bit depth: 8, yuv luminance scale: limited, matrix: 470bg".

yes, and the YUV420P8 is converted to RGB24. The main difference I see is that the source in the second script uses 0-255 as range while the second clip uses 16-235.
What happens if you use Levels to convert your input from 16-235 to 0-255 and adjust the range signaling to 'full' before applying DeOldify?
I also updated the test version with a version where Hybrid will for 'limited' content create code like this:

# changing range from limited to full range
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=0)
# adjusting color space from YUV420P8 to RGB24 for vsDeOldify
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="full")
# adding colors using DeOldify
from vsdeoldify import ddeoldify
clip = ddeoldify(clip=clip, method=0, deoldify_p=[024, 1, 0])
# changing range from full to limited range
clip = core.resize.Bicubic(clip, range_in_s="full", range_s="limited")

which converts limited to full before applying DeOldify.

Cu Selur

[Dan64]

Unfortunately this fix is not enough

Using your script I obtain the following image:




That is better but not equal to "native" RGB24 version

Using the following script

# Imports
import vapoursynth as vs
# getting Vapoursynth core
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'D:/Programs/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# loading plugins
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
# Import scripts
import validate
# Source: 'TestClipSingerBW.mp4'
# Current color space: YUV420P8, bit depth: 8, resolution: 480x360, frame rate: 25fps, scanorder: progressive, yuv luminance scale: limited, matrix: 470bg
# Loading TestClipSingerBW.mp4 using LWLibavSource
clip = core.lsmas.LWLibavSource(source="TestClipSingerBW.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=25, prefer_hw=0)
clip = core.std.Loop(clip=clip[185], times=257)
frame = clip.get_frame(0)
# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# setting color transfer (170), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip, _Transfer=6)
# setting color primaries info (to 470), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip, _Primaries=5)
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=0)
# making sure frame rate is set to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=0) # progressive
# adjusting color space from YUV420P8 to RGB24 for vsDeOldify
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="full")
# adding colors using DeOldify
from vsdeoldify import ddeoldify
clip = ddeoldify(clip=clip, method=0, deoldify_p=[0, 24, 1, 0])
# adjusting output color from: RGB24 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="full")
# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# output
clip.set_output()

 
In this case I removed the final conversion to limited, and the picture obtained is a little better



But to get the exact picture I need to used the same conditions as in the following script

# Imports
import vapoursynth as vs
# getting Vapoursynth core
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'D:/Programs/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# loading plugins
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
# Import scripts
import validate
# Source: 'TestClipSingerBW.mp4'
# Current color space: YUV420P8, bit depth: 8, resolution: 480x360, frame rate: 25fps, scanorder: progressive, yuv luminance scale: limited, matrix: 470bg
# Loading TestClipSingerBW.mp4 using LWLibavSource
clip = core.lsmas.LWLibavSource(source="TestClipSingerBW.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=25, prefer_hw=0)
clip = core.resize.Bicubic(clip=clip, matrix_in_s="470bg", matrix_s="709", range_in=0, range=0)
# changing range from limited to full range
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
clip = core.std.Loop(clip=clip[185], times=257)
frame = clip.get_frame(0)
# Setting detected color matrix (709).
clip = core.std.SetFrameProps(clip, _Matrix=1)
# setting color transfer (170), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip, _Transfer=1)
# setting color primaries info (to 709), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip, _Primaries=1)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=1)
# making sure frame rate is set to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=0) # progressive
# changing range from limited to full range
#clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=0)
# adjusting color space from YUV420P8 to RGB24 for vsDeOldify
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="709", range_s="full", dither_type="error_diffusion")
# adding colors using DeOldify
from vsdeoldify import ddeoldify
clip = ddeoldify(clip=clip, method=0, deoldify_p=[0, 24, 1, 0])
# changing range from full to limited range
#clip = core.resize.Bicubic(clip, range_in_s="full", range_s="limited")
# adjusting output color from: RGB24 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="709", range_s="full")
# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# output
clip.set_output()

In this case I obtain the following picture



To simplify the comparison I created this album: https://imgsli.com/MjYwNDcy

Dan

[Selur]

Man,.. why post the whole script, each time,..

Looking at you correct way:

# Source: 'TestClipSingerBW.mp4'
# Current color space: YUV420P8, bit depth: 8, resolution: 480x360, frame rate: 25fps, scanorder: progressive, yuv luminance scale: limited, matrix: 470bg
# Loading TestClipSingerBW.mp4 using LWLibavSource
clip = core.lsmas.LWLibavSource(source="TestClipSingerBW.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=25, prefer_hw=0)

load the source

clip = core.resize.Bicubic(clip=clip, matrix_in_s="470bg", matrix_s="709", range_in=0, range=0)

converts '470bg' to '709'

# changing range from limited to full range
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")

remaps 16-235 to 0-255

clip = core.std.Loop(clip=clip[185], times=257)

repeats frame 187, 257 times, no clue why you do that,...

# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=1)

this tells Hybrid the source is limted, this seems wrong, since you just converted it to 'full' beforehand.

# setting color range to PC (full) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=0)

So now you change set the color range to 'full' again

# adjusting color space from YUV420P8 to RGB24 for vsDeOldify
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="709", range_s="full", dither_type="error_diffusion")
# adding colors using DeOldify
from vsdeoldify import ddeoldify
clip = ddeoldify(clip=clip, method=0, deoldify_p=[0, 24, 1, 0])

now you apply DeOldify

So all in all up to this point (ignoring the frame duplication thing) you made sure that if YUV content was present it is converted to matrix bt709 and luma range full before converted to RGB and fed to DeOldify.

# adjusting output color from: RGB24 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="709", range_s="full")
# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# output
clip.set_output()

now you output full range.

Would have been way easier if you had spend the time to write down a few sentences of what you did or indented to do.

I write a test version which will make sure DeOldify is fed with 'bt709' and 'full' but will revert to the original values afterwards.
If you want 'bt709' and 'full' output, you should use Levels to convert the levels and ColorMatrix to change the matrix before feeding the source to DeOldify.

Cu Selur

[Dan64]

Hello Selur,

   I changed vsdeoldify to use internally the full range, previously vsdeoldify was using the limited range.
   This change together with your change, fixed the problem related to the color space conversion.

    You can find the comparison here: https://imgsli.com/MjYwNTM4

    Regarding the new version provided by you, a part the conversion of the range from "limited" to "full", I noted that you added the support to ColorTune.

    Unfortunately the implementation is not working perfectly. Main issues found are:

    1)  ColorTune is propagated to the custom parameters, but is not passed to the function ddeoldify_main()
    2) In some case ColorTune is not propagated correctly in the custom parameters (its destination is depending from selected preset)
    3)  ColorMap is not propagated correctly to custom parameters (its destination is depending from selected preset) and affects also ColorFix custom parameters

   I attached the new version with the support of "full range"

Dan

P.S.
In this release, in the clip description of function ddeoldify() I added the following advice: please remove any grain from the clip, because it will create artifacts on colored frames.

[Selur]

I changed vsdeoldify to use internally the full range, previously vsdeoldify was using the limited range.

Shouldn't it use the range it is fed with?!? (or throw an error if the luma range is different then required)


I will:
a. remove the full range requirement for the filter
b. look into fixing the problems with RC5

[Dan64]

    I changed vsdeoldify to use internally the full range, previously vsdeoldify was using the limited range.

Shouldn't it use the range it is fed with?!? (or throw an error if the luma range is different then required)


I will:
a. remove the full range requirement for the filter
b. look into fixing the problems with RC5

No, please the current version in my opinion is working perfectly, I expect to receive in input a source with matrix=709 and full range.

In the current version, at the beginning of vsdeoldify I'm using the following code

if clip.format.id != vs.RGB24:
    # clip not in RGB24 format, it will be converted
    if (clip.format.color_family == "YUV"):
        clip = clip.resize.Bicubic(format=vs.RGB24, matrix_in_s="709", range_s="full", dither_type="error_diffusion")
    else:
        clip = clip.resize.Bicubic(format=vs.RGB24, range_s="full")

  
Previously I was using

if clip.format.id != vs.RGB24:
   # clip not in RGB24 format, it will be converted
   if (clip.format.color_family == "YUV"):
      clip = clip.resize.Bicubic(format=vs.RGB24, matrix_in_s="709", range_s="limited", dither_type="error_diffusion")
    else:
      clip = clip.resize.Bicubic(format=vs.RGB24, range_s="limited")

Moreover, previously when It was necessary a change from RGB24 to YUV and then back to RGB24

I was a code like

clip = clip.resize.Bicubic(format=vs.YUV444PS, matrix_s="709", range_s="limited")
....
....
clip_rgb = clip.resize.Bicubic(format=vs.RGB24, matrix_in_s="709", range_s="limited", dither_type="error_diffusion")

now I'm using

clip = clip.resize.Bicubic(format=vs.YUV444PS, matrix_s="709", range_s="full")
...
...
clip_rgb = clip.resize.Bicubic(format=vs.RGB24, matrix_in_s="709", range_s="full", dither_type="error_diffusion")

So this version is expecting in input a RGB24 clip with matrix=709 and range full.

Dan

[Selur]

updated the test version (against R5), it should make sure it gets full&709

Cu Selur

[Dan64]

I still found the yesterday version

[Selur]

Argh, there's still a bug regarding the undoing of the color matrix change. (RGB doesn't have a color matrix entry)

[Selur]

Updated test download.