Selur's Little Message Board

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108

Okay, it did appear to good to be true. Smile

About the motion mask thing:

Code:
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")

core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/libmotionmask.dll")

import math

def vs_clip_color_stabilizer(clip: vs.VideoNode = None, nframes: int = 5, mode: str = 'center', scenechange: bool = True) -> vs.VideoNode:

    if nframes < 3 or nframes > 31: 

        raise ValueError("deoldify: number of frames must be in range: 3-31")

    if mode not in ['left', 'center', 'right']:

        raise ValueError("deoldify: mode must be 'left', 'center', or 'right'.")

    # convert the clip format for AverageFrames to YUV    

    clip_yuv = clip.resize.Bicubic(format=vs.YUV444PS, matrix_s="709", range_s="limited")

    if nframes%2==0:

        nframes +=1

    N = max(3, min(nframes, 31))

    Nh = round((N-1)/2)    

    Wi = math.trunc(100.0/N)

    if mode in ['left', 'right']:

        Wi = 2*Wi        

        Wc = 100-(Nh)*Wi

    else:

        Wc = 100-(N-1)*Wi

    weight_list = list()       

    for i in range(0, Nh):

        if mode in ['left', 'center']:

            weight_list.append(Wi)

        else:

            weight_list.append(0)

    weight_list.append(Wc)    

    for i in range(0, Nh):

        if mode in ['right', 'center']:    

            weight_list.append(Wi)

        else:

            weight_list.append(0)

    clip_yuv = vs.core.misc.AverageFrames(clip_yuv, weight_list, scenechange = True, planes=[1,2])           

    # convert the clip format for deoldify to RGB24 

    clip_rgb = clip_yuv.resize.Bicubic(format=vs.RGB24, matrix_in_s="709", range_s="limited", dither_type="error_diffusion") 

    return clip_rgb

# do filtering

# clip = deoldify ...

# do stabilization

clipFiltered = vs_clip_color_stabilizer(clip, nframes = 31)

# calculate motion mask on GrayImage

clipMask = clipFiltered 

clipMask = core.resize.Bicubic(clip=clipMask, format=vs.GRAY8, matrix_s="470bg", range_s="limited")

clipMask = core.motionmask.MotionMask(clip=clipMask, th1=10, th2=10, tht=15) # these are the defaults

clipMask = core.std.InvertMask(clipMask) # invert so that the not moving parts are white

# merge in YUV color space

clipMask = core.resize.Bicubic(clip=clipMask, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")

clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")

clip = core.resize.Bicubic(clip=clip, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")

clip = core.std.MaskedMerge(clip, clipFiltered, clipMask) # MotionMask

Cu Selur

Got a question:

Code:
clip = ddeoldify(clip=clip, model=0, render_factor=31, sat=[1.00,1], hue=[0.00,0], chroma_resize=False, color_stabilizer=[False,False,False,11,"center",True])

should only call deoldify, right?
Strange thing is, when I change:

Code:
if dd_method == 0 or (dd_method != 1):     

        clipa = clip.std.ModifyFrame(clip, ddeoldify_colorize)

Code:
if dd_method == 0 or (dd_method != 1):     

        return clip.std.ModifyFrame(clip, ddeoldify_colorize)

speed increases form 7.77fps to 9.24fps.
It seems like I overlooked something and not only simple deoldify is applied.
Any idea what is happening?

Cu Selur

I don't think that this is the problem

I attached a sample. So that you can replicate the results.

I simplified the script, so that the average is simply calculated as

Code:
clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False, planes=[1,2])

you can move the weight with value = 90, to see how each frame impact on the average.

No all frames have the same problem. The most interesting is the frame n.439 try to use it as reference.

good luck!

Dan

(20.03.2024, 18:50)Selur Wrote: [ -> ]Got a question:

Code:
clip = ddeoldify(clip=clip, model=0, render_factor=31, sat=[1.00,1], hue=[0.00,0], chroma_resize=False, color_stabilizer=[False,False,False,11,"center",True])
should only call deoldify, right?
Strange thing is, when I change:

Code:
if dd_method == 0 or (dd_method != 1): clipa = clip.std.ModifyFrame(clip, ddeoldify_colorize)
to

Code:
if dd_method == 0 or (dd_method != 1): return clip.std.ModifyFrame(clip, ddeoldify_colorize)
speed increases form 7.77fps to 9.24fps.
It seems like I overlooked something and not only simple deoldify is applied.
Any idea what is happening?

Cu Selur

In the script the default is dd_method=2, so in the first case you are using both the filters, in the second you are using only deoldify (not resized, because also chroma_resize=True by default).

No,...

Code:
clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])

Code:
clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False,

the code you send had both lines in it (none commented out)

try:

Code:
# Imports

import vapoursynth as vs

# getting Vapoursynth core

core = vs.core

# Loading Plugins

core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")

core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/libmotionmask.dll")

core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")

clip = core.lsmas.LWLibavSource(source="VideoTest8_small.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=24000, fpsden=1001, prefer_hw=0)

frame = clip.get_frame(0)

# Setting detected color matrix (709).

clip = core.std.SetFrameProps(clip, _Matrix=1)

# Setting color transfer (709), if it is not set.

if '_Transfer' not in frame.props or not frame.props['_Transfer']:

  clip = core.std.SetFrameProps(clip, _Transfer=1)

# Setting color primaries info (to 709), if it is not set.

if '_Primaries' not in frame.props or not frame.props['_Primaries']:

  clip = core.std.SetFrameProps(clip, _Primaries=1)

# Setting color range to TV (limited) range.

clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)

# making sure frame rate is set to 23.976

clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)

clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive

org = clip

#clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])

clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False, planes=[1,2])

# calculate motion mask on GrayImage

clipMask = clipFiltered

clipMask = core.resize.Bicubic(clip=clipMask, format=vs.GRAY8, matrix_s="470bg", range_s="limited")

clipMask = core.motionmask.MotionMask(clip=clipMask, th1=10, th2=10, tht=15) # these are the defaults

clipMask = core.std.InvertMask(clipMask) # invert so that the not moving parts are white

# merge in YUV color space

clipMask = core.resize.Bicubic(clip=clipMask, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")

clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")

clip = core.resize.Bicubic(clip=clip, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")

clip = core.std.MaskedMerge(clip, clipFiltered, clipMask) # MotionMask

org = core.resize.Bicubic(clip=org, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")

clip = core.std.StackVertical([org.text.Text(text="Orginal"),clipFiltered.text.Text(text="Averaged"),clip.text.Text(text="+MotionMask"), clipMask.text.Text(text="Mask")])

# set output frame rate to 23.976fps (progressive)

clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)

# Output

clip.set_output()

Cu Selur

(20.03.2024, 19:09)Selur Wrote: [ -> ]No,...

Code:
clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])

Code:
clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False,

the code you send had both lines in it (none commented out)

try this code with one line removed

Code:
# Imports

import vapoursynth as vs

# getting Vapoursynth core

core = vs.core

# Loading Plugins

core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/SharpenFilter/CAS/CAS.dll")

core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")

core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")

clip = core.lsmas.LWLibavSource(source="VideoTest8_small.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=24000, fpsden=1001, prefer_hw=0)

frame = clip.get_frame(0)

# Setting detected color matrix (709).

clip = core.std.SetFrameProps(clip, _Matrix=1)

# Setting color transfer (709), if it is not set.

if '_Transfer' not in frame.props or not frame.props['_Transfer']:

  clip = core.std.SetFrameProps(clip, _Transfer=1)

# Setting color primaries info (to 709), if it is not set.

if '_Primaries' not in frame.props or not frame.props['_Primaries']:

  clip = core.std.SetFrameProps(clip, _Primaries=1)

# Setting color range to TV (limited) range.

clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)

# making sure frame rate is set to 23.976

clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)

clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive

clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])

# set output frame rate to 23.976fps (progressive)

clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)

# Output

clip.set_output()

You will obtain the following frame

[Image: attachment.php?aid=2312]

It is the forward frame that is contributing the gray color.

(20.03.2024, 17:27)Selur Wrote: [ -> ]Problem is, that as soon as there is motion, understandably the averaging can cause problems.
I suspect that the idea of using a 'stupid' average is the main problem here.
=> the more motion the clip has the smaller the number of frames taken into account for the averaging need to be.
(maybe using a motion mask - created over the luma only - combined with AverageFrames - using the colored frames - could work, to only merge static chroma,..)

But removing from the stabilization the moving part you are also reducing the stabilization capability...

Dan

Not following you,..
I agree that:

Code:
clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2]

looks bad.
But:

Code:
clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False, planes=[1,2])

does not, to me it still seems that the farther you get away from the current frame (the middle) and the more the frames differ the more problems you get.
The extreme case is on a scene change,...

Cu Selur

Quote: But removing from the stabilization the moving part you are also reducing the stabilization capability...

Unless you segment the clip into sections and track the movement of the sections you can't avoid that.
=> AverageFrame is the wrong approach using the motion mask one can kind of remove some of the problem, but the fact is, that his approach has its flaws and probably can't solve the problem.

Cu Selur

I tested the motion mask approach on my reference clip that is plenty of psychedelic flashing colors, and unfortunately was unable to stabilize the clip.
The "AverageFrame" despite its defects was able to fix them, so I prefer to keep it.
In the new release I will reduce the max number of frames to 9 and will set the default value to 5.

Dan

Sounds good to me. Clips with lot of motion will still have issues, but I don't see any 'fast' solution for that.

Cu Selur

fixed the problem that dd_method wasn't shown unless ddcolor was used.
(updated download, same link)

Cu Selur

Selur

Selur

Dan64

Selur

Dan64

Selur

Selur

Dan64

Selur

Selur