Deoldify Vapoursynth filter

[Selur]

Okay, it did appear to good to be true.

About the motion mask thing:

core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/libmotionmask.dll")

import math
def vs_clip_color_stabilizer(clip: vs.VideoNode = None, nframes: int = 5, mode: str = 'center', scenechange: bool = True) -> vs.VideoNode:
    
    if nframes < 3 or nframes > 31:
        raise ValueError("deoldify: number of frames must be in range: 3-31")
        
    if mode not in ['left', 'center', 'right']:
        raise ValueError("deoldify: mode must be 'left', 'center', or 'right'.")
    
    # convert the clip format for AverageFrames to YUV    
    clip_yuv = clip.resize.Bicubic(format=vs.YUV444PS, matrix_s="709", range_s="limited")
    
    if nframes%2==0:
        nframes +=1
    
    N = max(3, min(nframes, 31))
    Nh = round((N-1)/2)    
    Wi = math.trunc(100.0/N)
    
    if mode in ['left', 'right']:
        Wi = 2*Wi        
        Wc = 100-(Nh)*Wi
    else:
        Wc = 100-(N-1)*Wi
    
    weight_list = list()      
    for i in range(0, Nh):
        if mode in ['left', 'center']:
            weight_list.append(Wi)
        else:
            weight_list.append(0)
    weight_list.append(Wc)    
    for i in range(0, Nh):
        if mode in ['right', 'center']:    
            weight_list.append(Wi)
        else:
            weight_list.append(0)
    
    clip_yuv = vs.core.misc.AverageFrames(clip_yuv, weight_list, scenechange = True, planes=[1,2])          
    
    # convert the clip format for deoldify to RGB24
    clip_rgb = clip_yuv.resize.Bicubic(format=vs.RGB24, matrix_in_s="709", range_s="limited", dither_type="error_diffusion")
    
    return clip_rgb
    
# do filtering
# clip = deoldify ...

# do stabilization
clipFiltered = vs_clip_color_stabilizer(clip, nframes = 31)

# calculate motion mask on GrayImage
clipMask = clipFiltered
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.GRAY8, matrix_s="470bg", range_s="limited")
clipMask = core.motionmask.MotionMask(clip=clipMask, th1=10, th2=10, tht=15) # these are the defaults
clipMask = core.std.InvertMask(clipMask) # invert so that the not moving parts are white
# merge in YUV color space
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.std.MaskedMerge(clip, clipFiltered, clipMask) # MotionMask

Cu Selur

[Selur]

Got a question:

clip = ddeoldify(clip=clip, model=0, render_factor=31, sat=[1.00,1], hue=[0.00,0], chroma_resize=False, color_stabilizer=[False,False,False,11,"center",True])

should only call deoldify, right?
Strange thing is, when I change:

if dd_method == 0 or (dd_method != 1):    
        clipa = clip.std.ModifyFrame(clip, ddeoldify_colorize)

to

if dd_method == 0 or (dd_method != 1):    
        return clip.std.ModifyFrame(clip, ddeoldify_colorize)

speed increases form 7.77fps to 9.24fps.
It seems like I overlooked something and not only simple deoldify is applied.
Any idea what is happening?

Cu Selur

[Dan64]

I don't think that this is the problem

I attached a sample. So that you can replicate the results.

I simplified the script, so that the average is simply calculated as

clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False, planes=[1,2])

you can move the weight with value = 90, to see how each frame impact on the average.

No all frames have the same problem. The most interesting is the frame n.439 try to use it as reference.

good luck!

Dan

---

Got a question:

clip = ddeoldify(clip=clip, model=0, render_factor=31, sat=[1.00,1], hue=[0.00,0], chroma_resize=False, color_stabilizer=[False,False,False,11,"center",True])

should only call deoldify, right?
Strange thing is, when I change:

    if dd_method == 0 or (dd_method != 1):     
        clipa = clip.std.ModifyFrame(clip, ddeoldify_colorize)

to

    if dd_method == 0 or (dd_method != 1):     
        return clip.std.ModifyFrame(clip, ddeoldify_colorize) 

speed increases form 7.77fps to 9.24fps.
It seems like I overlooked something and not only simple deoldify is applied.
Any idea what is happening?

Cu Selur

In the script the default is dd_method=2, so in the first case you are using both the filters, in the second you are using only deoldify (not resized, because also chroma_resize=True by default).

[Selur]

No,...

clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])

clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False,

the code you send had both lines in it (none commented out)

try:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/libmotionmask.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
clip = core.lsmas.LWLibavSource(source="VideoTest8_small.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=24000, fpsden=1001, prefer_hw=0)
frame = clip.get_frame(0)
# Setting detected color matrix (709).
clip = core.std.SetFrameProps(clip, _Matrix=1)
# Setting color transfer (709), if it is not set.
if '_Transfer' not in frame.props or not frame.props['_Transfer']:
  clip = core.std.SetFrameProps(clip, _Transfer=1)
# Setting color primaries info (to 709), if it is not set.
if '_Primaries' not in frame.props or not frame.props['_Primaries']:
  clip = core.std.SetFrameProps(clip, _Primaries=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 23.976
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
org = clip
#clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])
clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False, planes=[1,2])
# calculate motion mask on GrayImage
clipMask = clipFiltered
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.GRAY8, matrix_s="470bg", range_s="limited")
clipMask = core.motionmask.MotionMask(clip=clipMask, th1=10, th2=10, tht=15) # these are the defaults
clipMask = core.std.InvertMask(clipMask) # invert so that the not moving parts are white
# merge in YUV color space
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.std.MaskedMerge(clip, clipFiltered, clipMask) # MotionMask
org = core.resize.Bicubic(clip=org, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.std.StackVertical([org.text.Text(text="Orginal"),clipFiltered.text.Text(text="Averaged"),clip.text.Text(text="+MotionMask"), clipMask.text.Text(text="Mask")])


# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

Cu Selur

[Dan64]

No,...

clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])

clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False,

the code you send had both lines in it (none commented out)

try this code with one line removed

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/SharpenFilter/CAS/CAS.dll")
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
clip = core.lsmas.LWLibavSource(source="VideoTest8_small.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=24000, fpsden=1001, prefer_hw=0)
frame = clip.get_frame(0)
# Setting detected color matrix (709).
clip = core.std.SetFrameProps(clip, _Matrix=1)
# Setting color transfer (709), if it is not set.
if '_Transfer' not in frame.props or not frame.props['_Transfer']:
  clip = core.std.SetFrameProps(clip, _Transfer=1)
# Setting color primaries info (to 709), if it is not set.
if '_Primaries' not in frame.props or not frame.props['_Primaries']:
  clip = core.std.SetFrameProps(clip, _Primaries=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 23.976
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])
# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

You will obtain the following frame



It is the forward frame that is contributing the gray color.

---

Problem is, that as soon as there is motion, understandably the averaging can cause problems.
I suspect that the idea of using a 'stupid' average is the main problem here.
=> the more motion the clip has the smaller the number of frames taken into account for the averaging need to be.
(maybe using a motion mask - created over the luma only - combined with AverageFrames - using the colored frames - could work, to only merge static chroma,..)

But removing from the stabilization the moving part you are also reducing the stabilization capability...

Dan

[Selur]

Not following you,..
I agree that:

clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2]

looks bad.
But:

clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False, planes=[1,2])

does not, to me it still seems that the farther you get away from the current frame (the middle) and the more the frames differ the more problems you get.
The extreme case is on a scene change,...

Cu Selur

[Selur]

But removing from the stabilization the moving part you are also reducing the stabilization capability...

Unless you segment the clip into sections and track the movement of the sections you can't avoid that.
=> AverageFrame is the wrong approach using the motion mask one can kind of remove some of the problem, but the fact is, that his approach has its flaws and probably can't solve the problem.

Cu Selur

[Dan64]

I tested the motion mask approach on my reference clip that is plenty of psychedelic flashing colors, and unfortunately was unable to stabilize the clip.
The "AverageFrame" despite its defects was able to fix them, so I prefer to keep it. 
In the new release I will reduce the max number of frames to 9 and will set the default value to 5.

Dan

[Selur]

Sounds good to me. Clips with lot of motion will still have issues, but I don't see any 'fast' solution for that.

Cu Selur

[Selur]

fixed the problem that dd_method wasn't shown unless ddcolor was used.
(updated download, same link)

Cu Selur