This forum uses cookies
This forum makes use of cookies to store your login information if you are registered, and your last visit if you are not. Cookies are small text documents stored on your computer; the cookies set by this forum can only be used on this website and pose no security risk. Cookies on this forum also track the specific topics you have read and when you last read them. Please confirm whether you accept or reject these cookies being set.

A cookie will be stored in your browser regardless of choice to prevent you being asked this question again. You will be able to change your cookie settings at any time using the link in the footer.

Deoldify Vapoursynth filter
Okay, it did appear to good to be true. Smile

About the motion mask thing:
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/libmotionmask.dll")

import math
def vs_clip_color_stabilizer(clip: vs.VideoNode = None, nframes: int = 5, mode: str = 'center', scenechange: bool = True) -> vs.VideoNode:
    
    if nframes < 3 or nframes > 31:
        raise ValueError("deoldify: number of frames must be in range: 3-31")
        
    if mode not in ['left', 'center', 'right']:
        raise ValueError("deoldify: mode must be 'left', 'center', or 'right'.")
    
    # convert the clip format for AverageFrames to YUV    
    clip_yuv = clip.resize.Bicubic(format=vs.YUV444PS, matrix_s="709", range_s="limited")
    
    if nframes%2==0:
        nframes +=1
    
    N = max(3, min(nframes, 31))
    Nh = round((N-1)/2)    
    Wi = math.trunc(100.0/N)
    
    if mode in ['left', 'right']:
        Wi = 2*Wi        
        Wc = 100-(Nh)*Wi
    else:
        Wc = 100-(N-1)*Wi
    
    weight_list = list()      
    for i in range(0, Nh):
        if mode in ['left', 'center']:
            weight_list.append(Wi)
        else:
            weight_list.append(0)
    weight_list.append(Wc)    
    for i in range(0, Nh):
        if mode in ['right', 'center']:    
            weight_list.append(Wi)
        else:
            weight_list.append(0)
    
    clip_yuv = vs.core.misc.AverageFrames(clip_yuv, weight_list, scenechange = True, planes=[1,2])          
    
    # convert the clip format for deoldify to RGB24
    clip_rgb = clip_yuv.resize.Bicubic(format=vs.RGB24, matrix_in_s="709", range_s="limited", dither_type="error_diffusion")
    
    return clip_rgb
    
# do filtering
# clip = deoldify ...

# do stabilization
clipFiltered = vs_clip_color_stabilizer(clip, nframes = 31)

# calculate motion mask on GrayImage
clipMask = clipFiltered
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.GRAY8, matrix_s="470bg", range_s="limited")
clipMask = core.motionmask.MotionMask(clip=clipMask, th1=10, th2=10, tht=15) # these are the defaults
clipMask = core.std.InvertMask(clipMask) # invert so that the not moving parts are white
# merge in YUV color space
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.std.MaskedMerge(clip, clipFiltered, clipMask) # MotionMask

Cu Selur
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
Reply
Got a question:
clip = ddeoldify(clip=clip, model=0, render_factor=31, sat=[1.00,1], hue=[0.00,0], chroma_resize=False, color_stabilizer=[False,False,False,11,"center",True])
should only call deoldify, right?
Strange thing is, when I change:
if dd_method == 0 or (dd_method != 1):    
        clipa = clip.std.ModifyFrame(clip, ddeoldify_colorize)
to
if dd_method == 0 or (dd_method != 1):    
        return clip.std.ModifyFrame(clip, ddeoldify_colorize)
speed increases form 7.77fps to 9.24fps.
It seems like I overlooked something and not only simple deoldify is applied.
Any idea what is happening?

Cu Selur
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
Reply
I don't think that this is the problem

I attached a sample. So that you can replicate the results.

I simplified the script, so that the average is simply calculated as

clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False, planes=[1,2])

you can move the weight with value = 90, to see how each frame impact on the average.

No all frames have the same problem. The most interesting is the frame n.439 try to use it as reference.

good luck!

Dan

(20.03.2024, 18:50)Selur Wrote: Got a question:
clip = ddeoldify(clip=clip, model=0, render_factor=31, sat=[1.00,1], hue=[0.00,0], chroma_resize=False, color_stabilizer=[False,False,False,11,"center",True])
should only call deoldify, right?
Strange thing is, when I change:
    if dd_method == 0 or (dd_method != 1):     
        clipa = clip.std.ModifyFrame(clip, ddeoldify_colorize)
to
    if dd_method == 0 or (dd_method != 1):     
        return clip.std.ModifyFrame(clip, ddeoldify_colorize) 
speed increases form 7.77fps to 9.24fps.
It seems like I overlooked something and not only simple deoldify is applied.
Any idea what is happening?

Cu Selur

In the script the default is dd_method=2, so in the first case you are using both the filters, in the second you are using only deoldify (not resized, because also chroma_resize=True by default).


Attached Files
.zip   Test8.zip (Size: 1,93 MB / Downloads: 28)
Reply
No,...
clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])
[Image: grafik.png]
clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False,
[Image: grafik.png]

the code you send had both lines in it (none commented out)

try:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/libmotionmask.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
clip = core.lsmas.LWLibavSource(source="VideoTest8_small.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=24000, fpsden=1001, prefer_hw=0)
frame = clip.get_frame(0)
# Setting detected color matrix (709).
clip = core.std.SetFrameProps(clip, _Matrix=1)
# Setting color transfer (709), if it is not set.
if '_Transfer' not in frame.props or not frame.props['_Transfer']:
  clip = core.std.SetFrameProps(clip, _Transfer=1)
# Setting color primaries info (to 709), if it is not set.
if '_Primaries' not in frame.props or not frame.props['_Primaries']:
  clip = core.std.SetFrameProps(clip, _Primaries=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 23.976
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
org = clip
#clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])
clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False, planes=[1,2])
# calculate motion mask on GrayImage
clipMask = clipFiltered
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.GRAY8, matrix_s="470bg", range_s="limited")
clipMask = core.motionmask.MotionMask(clip=clipMask, th1=10, th2=10, tht=15) # these are the defaults
clipMask = core.std.InvertMask(clipMask) # invert so that the not moving parts are white
# merge in YUV color space
clipMask = core.resize.Bicubic(clip=clipMask, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.std.MaskedMerge(clip, clipFiltered, clipMask) # MotionMask
org = core.resize.Bicubic(clip=org, format=vs.YUV444PS, matrix_s="470bg", range_s="limited")
clip = core.std.StackVertical([org.text.Text(text="Orginal"),clipFiltered.text.Text(text="Averaged"),clip.text.Text(text="+MotionMask"), clipMask.text.Text(text="Mask")])


# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

Cu Selur
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
Reply
(20.03.2024, 19:09)Selur Wrote: No,...
clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])
[Image: grafik.png]
clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False,
[Image: grafik.png]

the code you send had both lines in it (none commented out)

try this code with one line removed
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/SharpenFilter/CAS/CAS.dll")
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
clip = core.lsmas.LWLibavSource(source="VideoTest8_small.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=24000, fpsden=1001, prefer_hw=0)
frame = clip.get_frame(0)
# Setting detected color matrix (709).
clip = core.std.SetFrameProps(clip, _Matrix=1)
# Setting color transfer (709), if it is not set.
if '_Transfer' not in frame.props or not frame.props['_Transfer']:
  clip = core.std.SetFrameProps(clip, _Transfer=1)
# Setting color primaries info (to 709), if it is not set.
if '_Primaries' not in frame.props or not frame.props['_Primaries']:
  clip = core.std.SetFrameProps(clip, _Primaries=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 23.976
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2])
# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

You will obtain the following frame

[Image: attachment.php?aid=2312]

It is the forward frame that is contributing the gray color.

(20.03.2024, 17:27)Selur Wrote: Problem is, that as soon as there is motion, understandably the averaging can cause problems.
I suspect that the idea of using a 'stupid' average is the main problem here.
=> the more motion the clip has the smaller the number of frames taken into account for the averaging need to be.
(maybe using a motion mask - created over the luma only - combined with AverageFrames - using the colored frames - could work, to only merge static chroma,..)

But removing from the stabilization the moving part you are also reducing the stabilization capability...

Dan


Attached Files Thumbnail(s)
   
Reply
Not following you,..
I agree that:
clipFiltered = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90], scale=100, scenechange = False, planes=[1,2]
looks bad.
But:
clip = core.misc.AverageFrames(clip, [0, 0, 0, 0, 0, 0, 0, 0, 0, 90, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], scale=100, scenechange = False, planes=[1,2])
does not, to me it still seems that the farther you get away from the current frame (the middle) and the more the frames differ the more problems you get.
The extreme case is on a scene change,...

Cu Selur
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
Reply
Quote: But removing from the stabilization the moving part you are also reducing the stabilization capability...
Unless you segment the clip into sections and track the movement of the sections you can't avoid that.
=> AverageFrame is the wrong approach using the motion mask one can kind of remove some of the problem, but the fact is, that his approach has its flaws and probably can't solve the problem.

Cu Selur
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
Reply
I tested the motion mask approach on my reference clip that is plenty of psychedelic flashing colors, and unfortunately was unable to stabilize the clip.
The "AverageFrame" despite its defects was able to fix them, so I prefer to keep it. 
In the new release I will reduce the max number of frames to 9 and will set the default value to 5.

Dan
Reply
Sounds good to me. Clips with lot of motion will still have issues, but I don't see any 'fast' solution for that.

Cu Selur
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
Reply
fixed the problem that dd_method wasn't shown unless ddcolor was used.
(updated download, same link)

Cu Selur
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)