ProPainter Vapoursynth filter

[Selur]

I will introduce the parameter frame width/height because the speed will decrease with the frame size and review the mask(s) management.

What should that parameter do?
Maybe, optionally to speed things up, something like Hybrids 'Only on Region'-filter option could be used (which uses animate) to only apply the filter to a specific region.

# adjusting color using Levels on YUV420P8 (8 bit)
def vsLevelsRegion1(clip,*args):
  return core.std.Levels(clip=clip, min_in=16, max_in=235, min_out=16, max_out=235, gamma=2.00, planes=[0])
end = clip.num_frames-1
mapping = [
  (0, end), [vsLevelsRegion1],
]
clip = animate.run(clip, mapping, selection=(50,60,96,58))

The selection parameter of animate, basically crops out a region and later puts it back in, so the region the filter gets applied to gets smaller.

Cu Selur

[Dan64]

This is a good idea, but it is necessary to find a way to insert propainter in the rendering chain.

clip -> crop_frame -> propaint(crop_frame) -> full_frame -> clip

you are suggesting to insert propainter in the place of vsLevelsRegion1 ?

Dan

[Selur]

That is what animate does.

you are suggesting to insert propainter in the place of vsLevelsRegion1 ?

yes, something like:

import animate
def vsAnimateRegion1(clip,*args):
  return propainter(clip, length=25, mask_path="running_car_mask.png", device_index=0, enable_fp16=True)
end = clip.num_frames-1
mapping = [
  (0, end), [vsAnimateRegion1],
]
clip = animate.run(clip, mapping, selection=(400,80,100,180))

should work. (This does not work, since the png is not 400x80.)

Hmm,... using the above with

return propainter(clip, length=25, mask_path="running_car_mask_cropped_400x80.png", device_index=0, enable_fp16=True) # image attached

Does not work. Not sure why.


Cu Selur

[Dan64]

I tested this script

from vspropainter import propainter
def vsPropainter(clip,*args):
  return propainter(clip, length=25, mask_path="running_car_mask_small.png", device_index=0, enable_fp16=True)
end = clip.num_frames-1
mapping = [
  (0, end), [vsPropainter],
]
clip = animate.run(clip, mapping, selection=(344,72,123,182))

with the attached mask but is not working

Dan

[Selur]

Yeah, I did the same. (see: above )

No clue either why it's not working. (I made sure to use a resolution that is mod16.)

Cu Selur

[Dan64]

I think that the filter to be able to remove the logo need a large context.
In the selected sample the logo is at the center of image, that is an optimal position for the filter.
Not sure if the filter will be able to remove the logo in the case is located for example at the bottom right of the frame.
I will test it.

Dan

[Selur]

Might be that there is a restriction between the resolution of the video and the amount of pixels that are marked for replacement.
i.e. replacement region (white part of the mask) needs to be less than 50% of the image or something like that.

[Dan64]

Try this example using the following command

clip = propainter(clip, length=250, mask_path="sample2_mask.png", enable_fp16=True, mask_dilation=16)

This time logo removal is good but not perfect.

Also the speed decreased to 0.40 fps, with the GPU memory peaked to 12145mb vs max memory of 12288mb.

Dan

[Selur]

Don't really see a difference with the mask dilation https://imgsli.com/MjY3ODgy, but the result looks fine in my book, given the (bad) quality of the source.
(The mask dilation should potentially increase the edge of the mask.)
Speed-wise with mask_dilation=16, I get:

encoded 235 frames in 107.66s (2.18 fps), 1811.52 kb/s, Avg QP:22.3

Cu Selur

Ps.: Wohaa,.. using a HD source eats ~32GB RAM and 15.2GB VRAM.

[Selur]

Okay, 1080p isn't usable atm. without something to limit the filtered size.
(45min and preview is still not visible)


Going to bed now.

Cu Selur