ProPainter Vapoursynth filter

[Dan64]

There is difference, see for example frame 74.

Here the comparison: https://imgsli.com/MjY3OTA5

using

clip = propainter(clip, length=100, mask_path="sample2_mask.png", enable_fp16=True, mask_dilation=8)

the speed increase to 1.11fps and the mask is big enough to cover all the edges.

Dan

---

By providing more frames, also the paper is rendered better: https://imgsli.com/MjY3OTE5

I attached RC2, with the color shift fixed.

Now the default is GPU=0 and f16 enabled.

Dan

[Selur]

in Lib\site-packages\vspropainter\model\misc.py", line 56

IS_HIGH_VERSION = [int(m) for m in list(re.findall(r"^([0-9]+)\.([0-9]+)\.([0-9]+)([^0-9][a-zA-Z0-9]*)?(\+git.*)?$",\

should be:

IS_HIGH_VERSION = [int(m) for m in list(re.findall(r"^(\d+)\.(\d+)\.(\d+)([\w\d\.].*)?$",\

to be 12.1
see: https://github.com/sczhou/CodeFormer/pul...2bb40536dc
(I did that before, but forgot to mention it)
With that adjustment, RC2 works as (adjusted) RC1 does here.

Cu Selur

[Selur]

Same test I ran in #9 is slower now.
Before I got:

encoded 192 frames in 50.79s (3.78 fps), 1707.74 kb/s, Avg QP:21.51

now I get:

encoded 192 frames in 87.40s (2.20 fps), 1707.83 kb/s, Avg QP:21.45

Okay, strangely commenting out the '@torch.inference_mode()'-lines gets speed up to 3.8fps again.
Side note: Additionally, increasing ref_stride to 100 increases speed to 3.99fps, increasing raft_iter to 30 does slow down the encoding to 2.02fps.

Keeping the '@torch.inference_mode()'-lines and using ref_stride=100 increases the speed to 5.14 fps.

Cu Selur

Ps.: did a small test with a non-transparent logo (see attachment)

[Selur]

The crop and only filter part of the image seems to work, but width&height need to be at least 256.

# Imports import vapoursynth as vs import site import sys import os # Initialize VapourSynth core core = vs.core # Import scripts folder scriptPath = 'F:/Hybrid/64bit/vsscripts' sys.path.insert(0, os.path.abspath(scriptPath)) # Load plugins core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll") # Import scripts import validate # Load source clip = core.lsmas.LWLibavSource(source=r"running_car.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=25, repeat=True, prefer_hw=0) # changing range from limited to full range for propainter clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full") # setting color range to PC (full) range. clip = core.std.SetFrameProps(clip=clip, _ColorRange=0) # adjusting color space from YUV420P8 to RGB24 for propainter clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="709", range_s="full") # Define the region coordinates and size x = 80 y = 70 width = 400 height = 256 # Create a blank clip for the mask of the same size as the original clip mask = core.std.BlankClip(clip=clip, color=[255, 255, 255]) mask = core.std.SetFrameProps(clip=mask, _ColorRange=1) mask = core.std.CropRel(mask, left=x, top=y, right=clip.width - (x + width), bottom=clip.height - (y + height)) mask = core.std.AddBorders(mask, left=x, top=y, right=clip.width - (x + width), bottom=clip.height - (y + height)) # Convert the mask to grayscale mask = core.resize.Bicubic(mask, format=vs.GRAY8, matrix_s="709") # Binarize Mask binarize_mask = core.std.BinarizeMask(clip, 1) # Crop to the region of interest cropped_clip = core.std.CropRel(clip, left=x, top=y, right=clip.width - (x + width), bottom=clip.height - (y + height)) # Apply propainter to the cropped region from vspropainter import propainter processed_cropped_clip = propainter(cropped_clip, length=250, mask_path="running_car_mask_cropped_400x256.png", device_index=0, enable_fp16=True) # Pad the processed region back to the original size padded_clip = core.std.AddBorders(processed_cropped_clip, left=x, top=y, right=clip.width - (x + width), bottom=clip.height - (y + height)) # Merge the processed region back into the original frame final_clip = core.std.MaskedMerge(clip, padded_clip, mask, planes=[0,1,2]) # undo range change final_clip = core.resize.Bicubic(final_clip, range_in_s="full", range_s="limited") # Adjust output color from RGB24 to YUV420P8 final_clip = core.resize.Bicubic(clip=final_clip, format=vs.YUV420P8, matrix_s="709", range_s="limited") # Set output frame rate to 25fps (progressive) final_clip = core.std.AssumeFPS(clip=final_clip, fpsnum=25, fpsden=1) # Output the final clip final_clip.set_output()

Speed increased to:

encoded 192 frames in 48.05s (4.00 fps), 1831.56 kb/s, Avg QP:21.5

Cu Selur

[Dan64]

in Lib\site-packages\vspropainter\model\misc.py", line 56

IS_HIGH_VERSION = [int(m) for m in list(re.findall(r"^([0-9]+)\.([0-9]+)\.([0-9]+)([^0-9][a-zA-Z0-9]*)?(\+git.*)?$",\

should be:

IS_HIGH_VERSION = [int(m) for m in list(re.findall(r"^(\d+)\.(\d+)\.(\d+)([\w\d\.].*)?$",\

to be 12.1
see: https://github.com/sczhou/CodeFormer/pul...2bb40536dc
(I did that before, but forgot to mention it)
With that adjustment, RC2 works as (adjusted) RC1 does here.

Cu Selur

Thanks for this fix, but the color fix was obtained by changing the following line

if convert_to_pil:             np_comp_frames = [Image.fromarray(cv2.cvtColor(cv2.resize(f, self.out_size), cv2.COLOR_BGR2RGB), 'RGB') for f in comp_frames]

in

if convert_to_pil:             np_comp_frames = [Image.fromarray(cv2.resize(f, self.out_size), 'RGB') for f in comp_frames]

in any case in the coming RC3 it will be not more necessary the conversion to PIL image

Dan

[Selur]

Yes, the above fix, is needed to be compatible with newer pytorch versions.

[Dan64]

I released the new RC3 (attached)

If are not found bugs should be also the initial release.

This is the (final) header:

def propainter(     clip: vs.VideoNode,     length: int = 100,     clip_mask: vs.VideoNode = None,     img_mask_path: str = None,     mask_dilation: int = 8,     neighbor_length: int = 10,     ref_stride: int = 10,     raft_iter: int = 20,     mask_region: tuple[int, int, int, int] = None,     weights_dir: str = model_dir,     enable_fp16: bool = True,     device_index: int = 0,     inference_mode: bool = False ) -> vs.VideoNode:     """ProPainter: Improving Propagation and Transformer for Video Inpainting     :param clip:            Clip to process. Only RGB24 format is supported.     :param length:          Sequence length that the model processes (min. 12 frames). High values will                             increase the inference speed but will increase also the memory usage. Default: 100     :param clip_mask:       Clip mask, must be of the same size and lenght of input clip. Default: None     :param img_mask_path:   Path of the mask image: Default: None     :param mask_dilation:   Mask dilation for video and flow masking. Default: 8     :param neighbor_length: Length of local neighboring frames. Low values decrease the                             memory usage. Default: 10     :param ref_stride:      Stride of global reference frames. High values will allow to                             reduce the memory usage and increase the inference speed. Default: 10     :param raft_iter:       Iterations for RAFT inference. Low values will decrease the inference                             speed but could affect the output quality. Default: 20     :param mask_region:     Allow to restirct the region of the mask, format: (width, height, left, top).                             The region must be big enough to allow the inference. Available only if clip_mask                             is specified. Default: None     :param enable_fp16:     If True use fp16 (half precision) during inference. Default: fp16 (for RTX30 or above)     :param device_index:    Device ordinal of the GPU (if = -1 CPU mode is enabled). Default: 0     :param inference_mode:  Enable/Disable torch inference mode. Default: False     """

  I added some interesting parameter.

1) clip_mask: now it is possible to pass a clip mask. You can test it using the provided sample and the following code:

# build clip mask clip_mask = core.imwri.Read(["running_car_mask.png"]) clip_mask = core.std.Loop(clip=clip_mask, times=clip.num_frames) clip_mask = core.std.AssumeFPS(clip=clip_mask, fpsnum=25, fpsden=1) # remove mask using propainter from vspropainter import propainter clip = propainter(clip, length=96, clip_mask=clip_mask)

In my tests it seems that using a mask clip even if there is only one frame, improves the speed.

2) mask_region: it is possible to define a smaller area for apply the propainter mask. You can test it using the provided sample and the following code:

# build clip mask clip_mask = core.imwri.Read(["running_car_mask.png"]) clip_mask = core.std.Loop(clip=clip_mask, times=clip.num_frames) clip_mask = core.std.AssumeFPS(clip=clip_mask, fpsnum=25, fpsden=1) # remove mask using propainter from vspropainter import propainter clip = propainter(clip, length=96, clip_mask=clip_mask, mask_region=(596-68*2, 336-28*2, 68, 28))

3) inference_mode: if true it will be enabled the "torch" inference mode. I don't have noted any speed increment/decrement by enabling it.

I was able to significantly speed up the inference using this code:

# build clip mask clip_mask = core.imwri.Read(["running_car_mask.png"]) clip_mask = core.std.Loop(clip=clip_mask, times=clip.num_frames) clip_mask = core.std.AssumeFPS(clip=clip_mask, fpsnum=25, fpsden=1) # remove mask using propainter from vspropainter import propainter clip = propainter(clip, length=96, clip_mask=clip_mask, mask_region=(596-88*2, 336-38*2, 88, 38), ref_stride=25, raft_iter=15, inference_mode=True)

 
have fun!

Dan

[Selur]

Nice. I'll do some testing.

Cu Selur

[Selur]

using:

clip = propainter(clip, clip_mask=clip_mask, length=96, mask_region=(596-88*2, 336-38*2, 88, 38), ref_stride=25, raft_iter=15, inference_mode=True)

I get:

encoded 192 frames in 39.53s (4.86 fps), 1645.06 kb/s, Avg QP:21.51

using:

clip = propainter(clip, clip_mask=clip_mask, length=250, mask_region=(596-88*2, 336-38*2, 88, 38), ref_stride=25, raft_iter=15, inference_mode=True)

I get:

encoded 192 frames in 38.54s (4.98 fps), 1649.41 kb/s, Avg QP:21.51

Nice. Not sure whether I get around to it today, but tomorrow I'll create a Hybrid dev version with support for ProPainter.
Will send you a link via pm once, I'm ready.

Cu Selur

[Dan64]

new version RC4. I added support for "mask_region" also for single image mask.

It is possible to test it using the following code:

from vspropainter import propainter clip = propainter(clip, length=96, img_mask_path="running_car_mask.png", mask_region=(596-68*2, 336-28*2, 68, 28), inference_mode=True)

Dan