Awfull clip from news broadcast fix

[georgepriftakis]

Thank you, I found what I was looking for my only problem now is that when I try to use spotless I get this
Failed to evaluate the script:
Python exception: MaskedMerge: mask clip must have same dimensions as main clip and be the same format or grayscale, passed YUV444P12[720x576] and Gray16[720x576]
Traceback (most recent call last):
File "src/cython/vapoursynth.pyx", line 3393, in vapoursynth._vpy_evaluate
File "src/cython/vapoursynth.pyx", line 3394, in vapoursynth._vpy_evaluate
File "C:\Users\gpriftakis\AppData\Local\Temp\tempPreviewVapoursynthFile15_04_58_716.vpy", line 81, in
clip = core.std.MaskedMerge(clip, clipFiltered, clipMask) # LimitMask
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "src/cython/vapoursynth.pyx", line 3129, in vapoursynth.Function.__call__
vapoursynth.Error: MaskedMerge: mask clip must have same dimensions as main clip and be the same format or grayscale, passed YUV444P12[720x576] and Gray16[720x576]

[Selur]

That's a bug, instead of YUV444P12 YUV444P16 would need to be used, or instead of GRAY16, GRAY12 would need to be used.

[Selur]

Uploaded a new dev version which should convert to GRAY12 and hopefully not break anything else.

Cu Selur

[didris]

Hi, Selur

I'm trying to remove a lot of scratches and white spots from a video, but I'm not having much success. If you have time, could you take a look at my script and suggest what could be improved? 

Thanks in advance.

I have uploaded the full video, if it is large I can cut out the part with the most problems:

https://www.swisstransfer.com/d/1bf1ee60-fc08-4b10-8824-be68141f0180

it's not the original video, I did some pre-processing with QTGMC from interlaced to progressive

import vapoursynth as vs
import ctypes

core = vs.core

# ---------------------------------------------------
# LOAD PLUGINS
# ---------------------------------------------------

core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/Support/akarin.dll")
core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/Support/DePan.dll")
core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/Support/vszip.dll")
core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/DenoiseFilter/ZSmooth/zsmooth.dll")
core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/DenoiseFilter/CTMF/CTMF.dll")
core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll")
core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/DenoiseFilter/KNLMeansCL/vsnlm_cuda.dll")
core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/DenoiseFilter/RemoveDirt/RemoveDirt.dll")
core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/FrameFilter/ReduceFlicker/ReduceFlicker.dll")
core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/ResizeFilter/nnedi3/vsznedi3.dll")
core.std.LoadPlugin("E:/Hybrid/64bit/vsfilters/GrainFilter/AddGrain/AddGrain.dll")

ctypes.windll.LoadLibrary(
    "E:/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll"
)

# ---------------------------------------------------
# IMPORTS
# ---------------------------------------------------

import dehalo
import removeDirt
import smdegrain
import stabilize
import SpotLess

# ---------------------------------------------------
# CLEANER SCRATCH MASK
# ---------------------------------------------------

def make_scratch_mask(clip):

    y = core.std.ShufflePlanes(
        clip,
        0,
        vs.GRAY
    )

    vert = core.std.Convolution(
        y,
        matrix=[
            -4, 0, 4,
            -12, 0, 12,
            -4, 0, 4
        ]
    )

    vert = core.std.Expr(
        vert,
        "x abs"
    )

    # safer threshold
    mask = core.std.Expr(
        vert,
        "x 9000 > 65535 0 ?"
    )

    # controlled expansion
    for _ in range(8):
        mask = core.std.Maximum(mask)

    for _ in range(5):
        mask = core.std.Inflate(mask)

    # softer blur
    mask = core.std.BoxBlur(
        mask,
        hradius=4,
        vradius=1
    )

    return mask

# ---------------------------------------------------
# TEMPORAL SCRATCH REPAIR
# ---------------------------------------------------

def repair_scratches(clip):

    y = core.std.ShufflePlanes(
        clip,
        0,
        vs.GRAY
    )

    mask = make_scratch_mask(clip)

    # previous / next frames
    prev = y[0] + y[:-1]
    next = y[1:] + y[-1]

    # temporal median rebuild
    repair = core.std.Expr(
        [prev, y, next],
        "x y z min max x z max min"
    )

    # light soften
    repair = core.std.BoxBlur(
        repair,
        hradius=3,
        vradius=1
    )

    fixed = core.std.MaskedMerge(
        y,
        repair,
        mask
    )

    return core.std.ShufflePlanes(
        [fixed, clip],
        [0, 1, 2],
        vs.YUV
    )

# ---------------------------------------------------
# PROCESS
# ---------------------------------------------------

def process(clip):

    # progressive
    clip = core.std.SetFrameProps(
        clip,
        _FieldBased=vs.FIELD_PROGRESSIVE
    )

    # ---------------------------------------------------
    # STABILIZE
    # ---------------------------------------------------

    clip = stabilize.Stab(
        clip,
        range=8,
        mirror=1
    )

    # ---------------------------------------------------
    # FLICKER
    # ---------------------------------------------------

    try:
        clip = core.reduceflicker.Reducer(
            clip,
            aggressive=1
        )
    except:
        pass

    # ---------------------------------------------------
    # LIGHT TEMPORAL CLEAN
    # ---------------------------------------------------

    clip = smdegrain.SMDegrain(
        clip,
        tr=2,
        thSAD=180,
        thSADC=120,
        interlaced=False,
        pel=2,
        subpixel=2
    )

    # ---------------------------------------------------
    # DIRT CLEAN
    # ---------------------------------------------------

    clip = SpotLess.SpotLess(
        clip,
        radT=3,
        asearch=5,
        thsad=600,
        thsad2=400,
        pel=2,
        smoother="zsmooth"
    )

    clip = removeDirt.RemoveDirtMC(
        clip,
        limit=16,
        gpu=False
    )

    # ---------------------------------------------------
    # SCRATCH REPAIR
    # ---------------------------------------------------

    clip = repair_scratches(clip)

    # ---------------------------------------------------
    # MAIN DENOISE
    # ---------------------------------------------------

    clip = smdegrain.SMDegrain(
        clip,
        tr=2,
        thSAD=220,
        thSADC=140,
        interlaced=False,
        pel=2,
        subpixel=2
    )

    # ---------------------------------------------------
    # FFT3D
    # ---------------------------------------------------

    clip = core.neo_fft3d.FFT3D(
        clip,
        sigma=2.0,
        sigma2=1.7,
        sigma3=1.4,
        sigma4=1.1,
        sharpen=0.0,
        planes=[0]
    )

    # ---------------------------------------------------
    # 16 BIT
    # ---------------------------------------------------

    clip = core.resize.Bicubic(
        clip,
        format=vs.YUV420P16
    )

    # ---------------------------------------------------
    # DEHALO
    # ---------------------------------------------------

    clip = dehalo.FineDehalo(
        clip,
        rx=2.4,
        ry=2.4,
        darkstr=0.0,
        brightstr=1.3
    )

    # ---------------------------------------------------
    # LIGHT NLMEANS
    # ---------------------------------------------------

    clip = core.nlm_cuda.NLMeans(
        clip,
        d=1,
        a=2,
        s=4,
        h=0.65,
        channels="Y"
    )

    # ---------------------------------------------------
    # SAFE SHARPEN
    # ---------------------------------------------------

    sharp = core.std.Expr(
        [clip, core.std.Maximum(clip)],
        "x 0.88 * y 0.12 * +"
    )

    clip = core.std.Merge(
        clip,
        sharp,
        weight=0.30
    )

    # ---------------------------------------------------
    # LIGHT GRAIN
    # ---------------------------------------------------

    clip = core.grain.Add(
        clip,
        var=0.08,
        uvar=0.03
    )

    # ---------------------------------------------------
    # FINAL OUTPUT
    # ---------------------------------------------------

    clip = core.resize.Bicubic(
        clip,
        format=vs.YUV420P10,
        dither_type="error_diffusion"
    )

    return clip

# ---------------------------------------------------
# ENTRY
# ---------------------------------------------------

clip = process(clip)
clip.set_output()

[Selur]

it's not the original video, I did some pre-processing with QTGMC from interlaced to progressive

Then me looking at it makes no sense, since removing artifacts that might have been in just one field isn't possible anymore.

Tip:
1. don't try using LLMs when trying to filter a source.
They can be helpful when writing functions where you know what they do, but they are bad ad tweaking or imagining scripts.

2. do "ctypes.windll.LoadLibrary" before loading plugins that may reply on the dlls, you load through windll

3. scratches are rarely just in the luma plane

Cu Selur


Ps.: since you are not using Hybrid, I'll move the thread to "A/V Talk"

[didris]

Hi, Selur

The original video: 

https://www.swisstransfer.com/d/0c1b89fe-e39e-4df5-a622-71839b527e18

I understand your point. My question was simply whether problems like those in this video can actually be corrected using Hybrid. If not, I’ll drop the idea.
If you’re able to help, I would be very grateful. If this thread is not useful or is causing unnecessary confusion, feel free to close it. Thanks for your time.

[Selur]

The mov isn't interlaced, it seems to be telecined and 12000/1001 fps
so I would use:

clip = core.tivtc.TFM(clip)
clip = core.tivtc.TDecimate(clip, mode=7, rate=11.9880, dupThresh=0.04, vidThresh=3.50, sceneThresh=15.00) # new fps: 11.988

or frame matching and frame reduction.

At a quick test I would add:

# Spot removal using SpotLess
clip = SpotLess.SpotLess(clip, radT=3, rec=True, pel=1, mStart=True, mEnd=True, iterations=2)

at it to remove most of the scratches&co
Then I would add DeNoise->QTGMC->InputType=2, for some more cleanup and anti-aiasing.

# Denoising using QTGMC
clip = qtgmc.QTGMC(clip, Preset="Fast", InputType=2, TR2=1, TFF=False, SourceMatch=0, Lossless=0, opencl=True)

Then I would think about replacing some of the frames with interpolations (wherever strong artifacts are that are in the surrounded by good frames). This should be done after the cleanup, so if you use Hybrid you might need to adjust the filter order for this.
If there are really ugly frames before or after a scene change I would add replace them with duplicates of the previous frame (using 'DuplicateFrames')

Additionally, I would throw some stabilization at it.

I would then play around with the setting of these filter to seem whether I can improve that.
Also, I would try throwing some machine learning stuff at it.

[didris]

Thank you very much, Selur
I will try according to your advice

[georgepriftakis]

Helo Selur.
I have a problem with another clip from the same show, those black lines on the clothes How can I remove them?


https://mega.nz/file/xV4G1LZJ#-Rv-akUeK0...QUI_ZLknN4

[Selur]

Did you already process the sample?
(looks over-sharpened and potentially resized or denoised)
At the current state, MCTemporalDenoise (veryhigh) should help somewhat.
If you have a version that is not pre-processed you have better changes to remove these properly. (might just be a broken field)

Cu Selur