ProPainter Vapoursynth filter

[Dan64]

I was not motivated in adding outpainting mode because on github page all the examples were on inpainting while on the project page on 16 clips only 1 was on outpainting.
So I removed the outpainting mode in the Vapoursynth version. 

If you are interested in having it, I can try to add it, let me know.

Dan

[Selur]

Yeah, if you find the time and motivation, outpainting would be nice.
Main usages for it that I see:
a. outpainting to do 4:3 <> 16:9 conversions (but that is probably too much to ask)
b. outpainting the edges of video content with distortions&co

Cu Selur

[Dan64]

Ps.: just updated the deoldify test download, since I noticed the 'Interference' didn't do a thing.

Not clear to what parameter you are referring to. This issue was already fixed in version "2024.06.28.1" ?

Dan

[Selur]

DOH,... mixed it up and meant ProPainter.
(wrote it correctly into the changelog)

[Dan64]

Anyway given the speed of ProPainter when is applied on small masks, probably the "outpainting" will be very slow...

I will try to add it.

Dan

[Selur]

Thanks for looking into it.

Cu Selur

[Dan64]

I attached the filter version with the support of outpainting and a demo for test.
As expected is not working, since cannot invent what has never seen.

Dan

[Selur]

Thanks, I'll give it a spin.

[Selur]

Using:

# Imports import vapoursynth as vs # getting Vapoursynth core import ctypes import sys import os core = vs.core # Import scripts folder scriptPath = 'F:/Hybrid/64bit/vsscripts' sys.path.insert(0, os.path.abspath(scriptPath)) # Loading Support Files Dllref = ctypes.windll.LoadLibrary("F:/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll") # loading plugins core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/GrainFilter/RemoveGrain/RemoveGrainVS.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/GrainFilter/AddGrain/AddGrain.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/EEDI3m_opencl.dll")# vsQTGMC core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/nnedi3/NNEDI3CL.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/libmvtools.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/scenechange.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/fmtconv.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/DeinterlaceFilter/Bwdif/Bwdif.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/DGDecNV/DGDecodeNV.dll") # Import scripts import havsfunc import validate # Source: 'G:\TestClips&Co\files\MPEG-2\Mächte des Wahnsinns.m2v' # Current color space: YUV420P8, bit depth: 8, resolution: 720x576, frame rate: 25fps, scanorder: top field first, yuv luminance scale: limited, matrix: 470bg, transfer: bt.470 system b/g, primaries: bt.601 pal # Loading G:\TestClips&Co\files\MPEG-2\Mächte des Wahnsinns.m2v using DGSource clip = core.dgdecodenv.DGSource("J:/tmp/m2v_10f10703815915d676a763d34caf8ab1_853323747.dgi",fieldop=0)# 25 fps, scanorder: top field first frame = clip.get_frame(0) # Setting detected color matrix (470bg). clip = core.std.SetFrameProps(clip=clip, _Matrix=5) # setting color transfer (601), if it is not set. if validate.transferIsInvalid(clip): clip = core.std.SetFrameProps(clip=clip, _Transfer=5) # setting color primaries info (to 470), if it is not set. if validate.primariesIsInvalid(clip): clip = core.std.SetFrameProps(clip=clip, _Primaries=5) # setting color range to TV (limited) range. clip = core.std.SetFrameProps(clip=clip, _ColorRange=1) # making sure frame rate is set to 25fps clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1) # making sure the detected scan type is set (detected: top field first) clip = core.std.SetFrameProps(clip=clip, _FieldBased=2) # tff original = clip # Deinterlacing using QTGMC clip = havsfunc.QTGMC(Input=clip, Preset="Fast", TFF=True, opencl=True) # new fps: 50 # Making sure content is preceived as frame based clip = core.std.SetFrameProps(clip=clip, _FieldBased=0) # progressive # Deinterlacing using QTGMC original = havsfunc.QTGMC(Input=original, Preset="Fast", TFF=True, opencl=True) # new fps: 50 # Making sure content is preceived as frame based original = core.std.SetFrameProps(clip=original, _FieldBased=0) # progressive clip = core.std.CropRel(clip=clip, left=2, right=0, top=68, bottom=70)# cropping to 718x438 original = core.std.CropRel(clip=original, left=2, right=0, top=68, bottom=70)# cropping to 718x438 # Resizing using 10 - bicubic spline clip = core.fmtc.resample(clip=clip, kernel="spline16", w=720, h=310, interlaced=False, interlacedd=False) # resolution 720x310 before YUV420P8 after YUV420P16 # changing range from limited to full range for vsProPainter clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full") # setting color range to PC (full) range. clip = core.std.SetFrameProps(clip=clip, _ColorRange=0) # adjusting color space from YUV420P8 to RGB24 for vsProPainter clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="709", range_s="full") # Inpainting using ProPainter from vspropainter import propainter_outpaint as ProPainterOutpaint clip = ProPainterOutpaint(clip=clip, length=80, outpaint_size=(720, 322)) # outpainting to 720x322 # internally changing color matrix for YUV<>RGB to '470bg' undoing color matrix change for vsProPainter # changing range from full to limited range for vsProPainter clip = core.resize.Bicubic(clip, range_in_s="full", range_s="limited") clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_s="limited", dither_type="error_diffusion") clip.set_output()

the output is 720x310 not 720x322.
Looking at the example:

clip = core.std.CropRel(clip=clip, left=0, right=0, top=52, bottom=52)# cropping to 720x300 ## changing color matrix from '470bg' to '709' for vsProPainter clip = core.resize.Bicubic(clip, matrix_in_s="470bg", matrix_s="709") # changing range from limited to full range for vsProPainter clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full") # setting color range to PC (full) range. clip = core.std.SetFrameProps(clip=clip, _ColorRange=0) # adjusting color space from YUV420P8 to RGB24 for vsProPainter clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="709", range_s="full") # Inpainting using ProPainter import vspropainter clip = vspropainter.propainter_outpaint(clip=clip, length=80, outpaint_size=(720, 340))

the output is 720x300. Same for the second exsample.

=> This does not seem correct.

Cu Selur

[Dan64]

attached version with correct resize.

Dan

---

The outpaint mode is based on the same code used for the inpainting.
In outpaint mode ProPainter builds automatically a mask on the edges of the clip and then try to remove it as in inpainting mode.

As example I atteched the runnicar demo, where there are 2 scripts:

• runcar_outpaint.vpy   : script using the new outpainting function
  
• runcar_inpaint.vpy      : script using the old inpainting function with a mask on the edges
  

both the scripts are providing the same results.

I hope that this can help in understanding better how ProPainter works.

Dan