ProPainter Vapoursynth filter

[Dan64]

I was not motivated in adding outpainting mode because on github page all the examples were on inpainting while on the project page on 16 clips only 1 was on outpainting.
So I removed the outpainting mode in the Vapoursynth version. 

If you are interested in having it, I can try to add it, let me know.

Dan

[Selur]

Yeah, if you find the time and motivation, outpainting would be nice.
Main usages for it that I see:
a. outpainting to do 4:3 <> 16:9 conversions (but that is probably too much to ask)
b. outpainting the edges of video content with distortions&co

Cu Selur

[Dan64]

Ps.: just updated the deoldify test download, since I noticed the 'Interference' didn't do a thing.

Not clear to what parameter you are referring to. This issue was already fixed in version "2024.06.28.1" ?

Dan

[Selur]

DOH,... mixed it up and meant ProPainter.
(wrote it correctly into the changelog)

[Dan64]

Anyway given the speed of ProPainter when is applied on small masks, probably the "outpainting" will be very slow...

I will try to add it.

Dan

[Selur]

Thanks for looking into it.

Cu Selur

[Dan64]

I attached the filter version with the support of outpainting and a demo for test.
As expected is not working, since cannot invent what has never seen.

Dan

[Selur]

Thanks, I'll give it a spin.

[Selur]

Using:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
import ctypes
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'F:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("F:/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll")
# loading plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/GrainFilter/RemoveGrain/RemoveGrainVS.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/GrainFilter/AddGrain/AddGrain.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/EEDI3m_opencl.dll")# vsQTGMC
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/nnedi3/NNEDI3CL.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/scenechange.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/DeinterlaceFilter/Bwdif/Bwdif.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/DGDecNV/DGDecodeNV.dll")
# Import scripts
import havsfunc
import validate
# Source: 'G:\TestClips&Co\files\MPEG-2\Mächte des Wahnsinns.m2v'
# Current color space: YUV420P8, bit depth: 8, resolution: 720x576, frame rate: 25fps, scanorder: top field first, yuv luminance scale: limited, matrix: 470bg, transfer: bt.470 system b/g, primaries: bt.601 pal
# Loading G:\TestClips&Co\files\MPEG-2\Mächte des Wahnsinns.m2v using DGSource
clip = core.dgdecodenv.DGSource("J:/tmp/m2v_10f10703815915d676a763d34caf8ab1_853323747.dgi",fieldop=0)# 25 fps, scanorder: top field first
frame = clip.get_frame(0)
# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip=clip, _Matrix=5)
# setting color transfer (601), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=5)
# setting color primaries info (to 470), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=5)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=1)
# making sure frame rate is set to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# making sure the detected scan type is set (detected: top field first)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=2) # tff
original = clip
# Deinterlacing using QTGMC
clip = havsfunc.QTGMC(Input=clip, Preset="Fast", TFF=True, opencl=True) # new fps: 50
# Making sure content is preceived as frame based
clip = core.std.SetFrameProps(clip=clip, _FieldBased=0) # progressive
# Deinterlacing using QTGMC
original = havsfunc.QTGMC(Input=original, Preset="Fast", TFF=True, opencl=True) # new fps: 50
# Making sure content is preceived as frame based
original = core.std.SetFrameProps(clip=original, _FieldBased=0) # progressive
clip = core.std.CropRel(clip=clip, left=2, right=0, top=68, bottom=70)# cropping to 718x438
original = core.std.CropRel(clip=original, left=2, right=0, top=68, bottom=70)# cropping to 718x438
# Resizing using 10 - bicubic spline
clip = core.fmtc.resample(clip=clip, kernel="spline16", w=720, h=310, interlaced=False, interlacedd=False) # resolution 720x310 before YUV420P8 after YUV420P16

# changing range from limited to full range for vsProPainter
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=0)
# adjusting color space from YUV420P8 to RGB24 for vsProPainter
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="709", range_s="full")
# Inpainting using ProPainter
from vspropainter import propainter_outpaint as ProPainterOutpaint
clip = ProPainterOutpaint(clip=clip, length=80, outpaint_size=(720, 322)) # outpainting to 720x322
# internally changing color matrix for YUV<>RGB to '470bg' undoing color matrix change for vsProPainter
# changing range from full to limited range for vsProPainter
clip = core.resize.Bicubic(clip, range_in_s="full", range_s="limited")
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_s="limited", dither_type="error_diffusion")

clip.set_output()

the output is 720x310 not 720x322.
Looking at the example:

clip = core.std.CropRel(clip=clip, left=0, right=0, top=52, bottom=52)# cropping to 720x300
## changing color matrix from '470bg' to '709' for vsProPainter
clip = core.resize.Bicubic(clip, matrix_in_s="470bg", matrix_s="709")
# changing range from limited to full range for vsProPainter
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=0)
# adjusting color space from YUV420P8 to RGB24 for vsProPainter
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="709", range_s="full")
# Inpainting using ProPainter
import vspropainter
clip = vspropainter.propainter_outpaint(clip=clip, length=80, outpaint_size=(720, 340))

the output is 720x300. Same for the second exsample.

=> This does not seem correct.

Cu Selur

[Dan64]

attached version with correct resize.

Dan

---

The outpaint mode is based on the same code used for the inpainting.
In outpaint mode ProPainter builds automatically a mask on the edges of the clip and then try to remove it as in inpainting mode.

As example I atteched the runnicar demo, where there are 2 scripts:

• runcar_outpaint.vpy   : script using the new outpainting function
  
• runcar_inpaint.vpy      : script using the old inpainting function with a mask on the edges
  

both the scripts are providing the same results.

I hope that this can help in understanding better how ProPainter works.

Dan