ProPainter Vapoursynth filter

[Dan64]

I was able to get this result with the attached script.



Let me know if is working on your side.

Dan

[Selur]

Nice!!
Yes, I can reproduce the effect with your script.
But why doesn't it also work when I use:

clip = ProPainter(clip, img_mask_path="C:/Users/Selur/Desktop/mask.png", length=50, mask_dilation=0, neighbor_length=4, mask_region=(1080,256,0,1300))

When I use your mask it works, but why does it not work with mine?
Your mask doesn't have the same resolution as the video, and it also doesn't even have the same display aspect ratio (340x600 vs 1080x1920)

Cu Selur

[Selur]

DOH, your source video also has a lower resolution, so the problem is probably related to vram consumption.

[Selur]

Okay, using the low resolution mask on my full resolution source, even when not lowering dilation, neighbor length, and length:

clip = ProPainter(clip, img_mask_path="C:/Users/Selur/Desktop/Education_ProPainter_small/mask_small.jpg", mask_region=(1080,256,0,1300))

does work.

=> this seems to be an issue with the mask handling
Does it make sense to add an option to vs-propainter to divide mask width&height by a factor X ?

Cu Selur

[Dan64]

I debugged the source and it seems that sometimes (depending on the input mask) the call to the function:

scipy.ndimage.binary_dilation(mask_img, iterations=mask_dilates)

 provides a wrong dilated mask. Not clear why and the debugging of "scipy" function is outside my scope.

So in summary if you get wrong results try to set 

mask_dilation=0       # skip the call to scipy.ndimage.binary_dilation()

   
 and check if this solve the problem.

Dan

P.S.
If you provide as input a low resolution mask, the mask is resized to the frame dimension inside the filter. 
In my tests mask_dilation=0 fixed the issue, but I worked on a small size clip.
For sure there is a problem on mask handling, probably due to memory constrains and using a mask region is the optimal way to manage it because the inference is performed only on the clip having the size of mask region.

[Selur]

using:

clip = ProPainter(clip, img_mask_path="C:/Users/Selur/Desktop/mask.png", length=12, mask_dilation=0, mask_region=(1080,256,0,1300))

It's still broken:

same with:

clip = ProPainter(clip, img_mask_path="C:/Users/Selur/Desktop/mask.png", length=12, mask_dilation=0, neighbor_length=4, mask_region=(1080,256,0,1300))

so it's not just fault of the dilation.
using:

clip = ProPainter(clip, img_mask_path="C:/Users/Selur/Desktop/mask.png", length=12, mask_dilation=0, neighbor_length=4)

Converting the PNG to jpg (4:2:0):

clip = ProPainter(clip, img_mask_path="C:/Users/Selur/Desktop/mask.jpg", length=12, mask_dilation=0, neighbor_length=4, mask_region=(1080,256,0,1300))

Converting the PNG to jpg (4:4:4 max quality):

clip = ProPainter(clip, img_mask_path="C:/Users/Selur/Desktop/mask.jpg", length=12, mask_dilation=0, neighbor_length=4, mask_region=(1080,256,0,1300))

so doesn't seem to be a jpg vs png problem.

Cu Selur

Ps.: going to bed now, but I can do more testing, tomorrow

[Dan64]

In the script "propainter_render.py", there is this code

def model_init(self, torch_device, model_dir: str = None, mpath: str = None):
        self.device = torch_device

        self.img_mask_is_cropped = False

        # load image mask if available
        if mpath is None:
            self.img_mask = None
        else:
            img_m = Image.open(mpath).convert('RGB')
            out_size = img_m.size
            if out_size == self.clip_size:
                self.img_mask = img_m
            else:
                self.img_mask = img_m.resize(self.clip_size)
        self.img_mask_orig = self.img_mask

So if you provide in input mask_small.jpg is resized as the image attached.

So looking to the code there is not difference in providing in input "mask_small.jpg" or "mask_big.png".

But using  "mask_small.jpg" it works while using "mask_big.png" it doesn't work, is very crazy, for the moment I don't have an explanation for this kind of odd behavior.

Dan