ProPainter Vapoursynth filter

[jamesharding26]

I realise now that you have it actively working in Hybrid, is there any chance I could have access to a dev where it's operating? I tried the latest dev in your experimental folder but cannot find propainter in the 'other' tab. Sorry if I've missed something obvious.

[Selur]

You, just need the normal version with the torch-addon. (linked on the download page)

Cu Selur

[Dan64]

Hi all,

I am trying to run the script and running into an error that I don't understand, if anyone can help that would be much appreciated

Failed to evaluate the script:
Python exception: list index out of range
Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3233, in vapoursynth._vpy_evaluate
File "src\\cython\\vapoursynth.pyx", line 3234, in vapoursynth._vpy_evaluate
File "", line 11, in
File "C:\Users\James\AppData\Local\Programs\Python\Python312\Lib\site-packages\vspropainter\__init__.py", line 24, in
from vspropainter.propainter_render import ModelProPainterIn, ModelProPainterOut
File "C:\Users\James\AppData\Local\Programs\Python\Python312\Lib\site-packages\vspropainter\propainter_render.py", line 22, in
from vspropainter.model.recurrent_flow_completion import RecurrentFlowCompleteNet
File "C:\Users\James\AppData\Local\Programs\Python\Python312\Lib\site-packages\vspropainter\model\recurrent_flow_completion.py", line 7, in
from vspropainter.model.misc import constant_init
File "C:\Users\James\AppData\Local\Programs\Python\Python312\Lib\site-packages\vspropainter\model\misc.py", line 57, in
IS_HIGH_VERSION = [int(m) for m in list(re.findall(r"^(\d+)\.(\d+)\.(\d+)([\w\d\.].*)?$",\
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: list index out of range

Thanks

James

This was a change applied by Selur in this post: #22
I think that Selur can suggest a fix for this issue.

Dan

[Selur]

Ahh,... that was needed since the way pytorch changed how the version was encoded.
This should work with new pytorch versions 2.6.* for older version your probably need to change the code, see: https://github.com/sczhou/CodeFormer/pul...2bb40536dc

Cu Selur

[Dan64]

If you are using Windows, it is safe to edit the file:

"C:\Users\James\AppData\Local\Programs\Python\Python312\Lib\site-packages\vspropainter\model\misc.py", line 57

and set

IS_HIGH_VERSION = False

When IS_HIGH_VERSION  is True the filter check if is available the mps backend, but this backend is useful only on MacOS, so if you are using Windows you can skip this check.

Dan

[Dan64]

Released a new version that should fix this issue: https://github.com/dan64/vs-propainter/r...ag/v1.12.1


Dan

[Selur]

@Dan64: I'm trying to remove the logo mentioned here of this clip.
I created a video mask
Code I used to create the logo:

clip = clip.resize.Bicubic(format=vs.RGBS)

# Extract R, G, B planes
r = clip.std.ShufflePlanes(0, vs.GRAY)
g = clip.std.ShufflePlanes(1, vs.GRAY)
b = clip.std.ShufflePlanes(2, vs.GRAY)

# Build an expression that detects yellow-ish pixels:
# - R and G are both high (>= 0.6)
# - B is relatively low (<= 0.5)
expr = "x 0.6 > y 0.6 > and z 0.5 < and x y - abs 0.1 < and 1 0 ?"
mask = core.std.Expr([r, g, b], expr=[expr])

# Clean up the mask
clip = mask.std.Inflate().std.Minimum().std.Minimum().std.Convolution(matrix=[1]*9).std.Maximum().std.Maximum()
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT709)
# colorformat GRAYS

which seems fine covering the logo for most of the clip.
Problem is when using:

# Inpainting using ProPainter
from vspropainter import propainter as ProPainter
clipMask = core.lsmas.LWLibavSource(source="G:/Output/logo_mask2.mp4", format="RGB24", cache=0)
clip = ProPainter(clip=clip, clip_mask=clipMask, length=500, neighbor_length=20, raft_iter=10, mask_region=(500,256,390,240))

starting with frame 149 the logo reappears (isn't properly removed).

the mask clip properly covers the logo:

If I reduce the length to 20, logo is gone until 160,...
I first thought this could this be a bug in the length&co parameter handling, for the edge case where the end of the clip is approached, in the wrapper.
To test my theory, I added clip + core.std.Reverse(clip).std.Trim(1), created a new mask clip and used that to look at frame 149+.
That didn't help, it even seemed to make it worse. Then I thought maybe the scene threshold was the issue, so I set that to 0, but that also didn't help.
Last thing I tried was disabling the mask region, but that also didn't really help either.
Do you think this is a problem of:
a. finding the right parameters
b. a bug in the wrapper
c. ProPainter just strangely failing
?

Cu Selur

[Dan64]

Hello Selur,

  theoretically clip mask should work (see attached sample), but I never used it on long clips.
  It seems that ProPainter is not able to manage properly the clip mask when it becomes almost static.
  You can try to split the movie in 2 parts, the moving part (till frame 99) and the static part (from 100 till 202).
  It could be also possible that ProPainter is not able to properly manage moving masks when are not available enough frames to perform the inference.
  In effect in the attached sample the player is simply going from right to left and then form left to right and in this case the inference is very simple.

Dan

[Dan64]

It seems that the problem is on the clip mask.

If you extract the frame 165 from the mask and copy it as transparent selection over the frame 165 rendered by ProPainter.

You obtain the following frame.




As you can see the letter I of the mask is over the letter F the rendered image. Try to check the coordinates they are different.

For the tests is better to don't use the crop region and use the full frame, but with half size, something like this

clipMask = clipMask.resize.Spline36(width=640, height=360)
clip = clip.resize.Spline36(width=640, height=360)
clip = ProPainter(clip=clip, clip_mask=clipMask, length=202)

Dan

[Selur]

That makes sense, I did not take the future masking into account => so I will have to look into a better method to create the masking clip. Thanks!

Cu Selur