Deoldify Vapoursynth filter - Printable Version

Deoldify Vapoursynth filter - Printable Version

+- Selur's Little Message Board (https://forum.selur.net)
+-- Forum: Talk, Talk, Talk (https://forum.selur.net/forum-5.html)
+--- Forum: Small Talk (https://forum.selur.net/forum-7.html)
+--- Thread: Deoldify Vapoursynth filter (/thread-3595.html)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143

RE: Deoldify Vapoursynth filter - Selur - 20.03.2024

Okay, it did appear to good to be true. Smile

About the motion mask thing:

core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/libmotionmask.dll")

import math
def vs_clip_color_stabilizer(clip: vs.VideoNode = None, nframes: int = 5, mode: str = 'center', scenechange: bool = True) -> vs.VideoNode:
    
    if nframes < 3 or nframes > 31: 
        raise ValueError("deoldify: number of frames must be in range: 3-31")
        
    if mode not in ['left', 'center', 'right']:
        raise ValueError("deoldify: mode must be 'left', 'center', or 'right'.")
    
    # convert the clip format for AverageFrames to YUV    
    clip_yuv = clip.resize.Bicubic(format=vs.YUV444PS, matrix_s="709", range_s="limited")
    
    if nframes%2==0:
        nframes +=1
    
    N = max(3, min(nframes, 31))
    Nh = round((N-1)/2)    
    Wi = math.trunc(100.0/N)
    
    if mode in ['left', 'right']:
        Wi = 2*Wi        
        Wc = 100-(Nh)*Wi
    else:
        Wc = 100-(N-1)*Wi
    
    weight_list = list()       
    for i in range(0, Nh):
        if mode in ['left', 'center']:
            weight_list.append(Wi)
        else:
            weight_list.append(0)
    weight_list.append(Wc)    
    for i in range(0, Nh):
        if mode in ['right', 'center']:    
            weight_list.append(Wi)
        else:
            weight_list.append(0)
    
    clip_yuv = vs.core.misc.AverageFrames(clip_yuv, weight_list, scenechange = True, planes=[1,2])           
    
    # convert the clip format for deoldify to RGB24 
    clip_rgb = clip_yuv.resize.Bicubic(format=vs.RGB24, matrix_in_s="709", range_s="limited", dither_type="error_diffusion") 
    
    return clip_rgb
    
# do filtering
# clip = deoldify ...

# do stabilization
clipFiltered = vs_clip_color_stabilizer(clip, nframes = 31)

# calculate motion mask on GrayImage
clipMask = clipFiltered 
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.GRAY8, matrix_s="470bg", range_s="limited")
clipMask = core.motionmask.MotionMask(clip=clipMask, th1=10, th2=10, tht=15) # these are the defaults
clipMask = core.std.InvertMask(clipMask) # invert so that the not moving parts are white
# merge in YUV color space
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.std.MaskedMerge(clip, clipFiltered, clipMask) # MotionMask

Cu Selur

RE: Deoldify Vapoursynth filter - Selur - 20.03.2024

Got a question:

clip = ddeoldify(clip=clip, model=0, render_factor=31, sat=[1.00,1], hue=[0.00,0], chroma_resize=False, color_stabilizer=[False,False,False,11,"center",True])

should only call deoldify, right?
Strange thing is, when I change:

if dd_method == 0 or (dd_method != 1):     
        clipa = clip.std.ModifyFrame(clip, ddeoldify_colorize)

if dd_method == 0 or (dd_method != 1):     
        return clip.std.ModifyFrame(clip, ddeoldify_colorize)

speed increases form 7.77fps to 9.24fps.
It seems like I overlooked something and not only simple deoldify is applied.
Any idea what is happening?

Cu Selur

RE: Deoldify Vapoursynth filter - Dan64 - 20.03.2024

I don't think that this is the problem

I attached a sample. So that you can replicate the results.

I simplified the script, so that the average is simply calculated as

clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False, planes=[1,2])

you can move the weight with value = 90, to see how each frame impact on the average.

No all frames have the same problem. The most interesting is the frame n.439 try to use it as reference.

good luck!

Dan

(20.03.2024, 18:50)Selur Wrote: Got a question:

clip = ddeoldify(clip=clip, model=0, render_factor=31, sat=[1.00,1], hue=[0.00,0], chroma_resize=False, color_stabilizer=[False,False,False,11,"center",True])
should only call deoldify, right?
Strange thing is, when I change:

if dd_method == 0 or (dd_method != 1): clipa = clip.std.ModifyFrame(clip, ddeoldify_colorize)
to

if dd_method == 0 or (dd_method != 1): return clip.std.ModifyFrame(clip, ddeoldify_colorize)
speed increases form 7.77fps to 9.24fps.
It seems like I overlooked something and not only simple deoldify is applied.
Any idea what is happening?

Cu Selur

In the script the default is dd_method=2, so in the first case you are using both the filters, in the second you are using only deoldify (not resized, because also chroma_resize=True by default).

RE: Deoldify Vapoursynth filter - Selur - 20.03.2024

No,...

clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])

clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False,

the code you send had both lines in it (none commented out)

try:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/libmotionmask.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
clip = core.lsmas.LWLibavSource(source="VideoTest8_small.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=24000, fpsden=1001, prefer_hw=0)
frame = clip.get_frame(0)
# Setting detected color matrix (709).
clip = core.std.SetFrameProps(clip, _Matrix=1)
# Setting color transfer (709), if it is not set.
if '_Transfer' not in frame.props or not frame.props['_Transfer']:
  clip = core.std.SetFrameProps(clip, _Transfer=1)
# Setting color primaries info (to 709), if it is not set.
if '_Primaries' not in frame.props or not frame.props['_Primaries']:
  clip = core.std.SetFrameProps(clip, _Primaries=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 23.976
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
org = clip
#clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])
clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False, planes=[1,2])
# calculate motion mask on GrayImage
clipMask = clipFiltered
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.GRAY8, matrix_s="470bg", range_s="limited")
clipMask = core.motionmask.MotionMask(clip=clipMask, th1=10, th2=10, tht=15) # these are the defaults
clipMask = core.std.InvertMask(clipMask) # invert so that the not moving parts are white
# merge in YUV color space
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.std.MaskedMerge(clip, clipFiltered, clipMask) # MotionMask
org = core.resize.Bicubic(clip=org, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.std.StackVertical([org.text.Text(text="Orginal"),clipFiltered.text.Text(text="Averaged"),clip.text.Text(text="+MotionMask"), clipMask.text.Text(text="Mask")])


# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

Cu Selur

RE: Deoldify Vapoursynth filter - Dan64 - 20.03.2024

(20.03.2024, 19:09)Selur Wrote: No,...

clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])

clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False,

the code you send had both lines in it (none commented out)

try this code with one line removed

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/SharpenFilter/CAS/CAS.dll")
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
clip = core.lsmas.LWLibavSource(source="VideoTest8_small.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=24000, fpsden=1001, prefer_hw=0)
frame = clip.get_frame(0)
# Setting detected color matrix (709).
clip = core.std.SetFrameProps(clip, _Matrix=1)
# Setting color transfer (709), if it is not set.
if '_Transfer' not in frame.props or not frame.props['_Transfer']:
  clip = core.std.SetFrameProps(clip, _Transfer=1)
# Setting color primaries info (to 709), if it is not set.
if '_Primaries' not in frame.props or not frame.props['_Primaries']:
  clip = core.std.SetFrameProps(clip, _Primaries=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 23.976
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])
# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

You will obtain the following frame

[Image: attachment.php?aid=2312]

It is the forward frame that is contributing the gray color.

(20.03.2024, 17:27)Selur Wrote: Problem is, that as soon as there is motion, understandably the averaging can cause problems.
I suspect that the idea of using a 'stupid' average is the main problem here.
=> the more motion the clip has the smaller the number of frames taken into account for the averaging need to be.
(maybe using a motion mask - created over the luma only - combined with AverageFrames - using the colored frames - could work, to only merge static chroma,..)

But removing from the stabilization the moving part you are also reducing the stabilization capability...

Dan

RE: Deoldify Vapoursynth filter - Selur - 20.03.2024

Not following you,..
I agree that:

clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2]

looks bad.
But:

clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False, planes=[1,2])

does not, to me it still seems that the farther you get away from the current frame (the middle) and the more the frames differ the more problems you get.
The extreme case is on a scene change,...

Cu Selur

RE: Deoldify Vapoursynth filter - Selur - 20.03.2024

Quote: But removing from the stabilization the moving part you are also reducing the stabilization capability...

Unless you segment the clip into sections and track the movement of the sections you can't avoid that.
=> AverageFrame is the wrong approach using the motion mask one can kind of remove some of the problem, but the fact is, that his approach has its flaws and probably can't solve the problem.

Cu Selur

RE: Deoldify Vapoursynth filter - Dan64 - 20.03.2024

I tested the motion mask approach on my reference clip that is plenty of psychedelic flashing colors, and unfortunately was unable to stabilize the clip.
The "AverageFrame" despite its defects was able to fix them, so I prefer to keep it.
In the new release I will reduce the max number of frames to 9 and will set the default value to 5.

Dan

RE: Deoldify Vapoursynth filter - Selur - 20.03.2024

Sounds good to me. Clips with lot of motion will still have issues, but I don't see any 'fast' solution for that.

Cu Selur

RE: Deoldify Vapoursynth filter - Selur - 20.03.2024

fixed the problem that dd_method wasn't shown unless ddcolor was used.
(updated download, same link)

Cu Selur