ProPainter Vapoursynth filter

[jamesharding26]

Hi everyone,

I'm having an issue getting Mask Region and Debug to work. When enabling debug, nothing changes and ProPainter still attempts to complete the work without showing me the mask region. And when I set a mask region, equally it doesn't seem to be used as the time to process is still exactly the same. Is anyone else seeing this issue?

[Selur]

Seems to work fine here:

[Selur]

Seems like 'img_mask_path' doesn't like whitespaces.

clip = ProPainter(clip, img_mask_path="G:/clips/logo samples/3. Lance Stroll Original HD Interrupts and Hits The Barrier 2021 F1 Monaco GP.png", length=500, mask_region=(512,256,1400,0))

fails with

"propainter: wrong image mask: " + img_mask_path

clip = ProPainter(clip, img_mask_path="G:/clips/logo samples/mask2.png", length=500, mask_region=(512,256,1400,0))

works.

Also: Can you add a min_scene_size parameter, so that scenes are at least min_scene_size frames long?
The clip from https://forum.videohelp.com/threads/4193...ost2786364 can't be handled otherwise, since there are a lot of flashes. (disabling scene change does work for the clip, but having a minimum might be better)


Cu Selur

[Dan64]

Hello Selur,

   I attached the RC1, where are applied the following changes:

      - improved image files identification
      - added new parameter sc_min_freq, if > 0 will be generated a scene change at least every "sc_min_freq" frames.


   If you need other changes please clarify better your needs.

Dan

[Dan64]

Supposing that your request was to have a parameter that will allow to set the minimum number of frames that must elapse between two accepted scene changes.
I attached the new RC2, where now the parameter sc_min_freq will allow to set this value. 
The new header of the function propainter_inpaint() is

def propainter_inpaint(
        clip: vs.VideoNode,
        length: int = 100,
        clip_mask: vs.VideoNode = None,
        img_mask_path: str = None,
        mask_dilation: int = 8,
        neighbor_length: int = 10,
        ref_stride: int = 10,
        raft_iter: int = 20,
        mask_region: tuple[int, int, int, int] = None,
        sc_threshold: float = 0.1,
        sc_min_freq: int = 0,
        weights_dir: str = model_dir,
        enable_fp16: bool = True,
        device_index: int = 0,
        inference_mode: bool = False
) -> vs.VideoNode:
    """ProPainter: Improving Propagation and Transformer for Video Inpainting

    :param clip:            Clip to process. Only RGB24 "full range" format is supported.
    :param length:          Sequence length that the model processes (min. 12 frames). High values will
                            increase the inference speed but will increase also the memory usage. Default: 100
    :param clip_mask:       Clip mask, must be of the same size and length of input clip. Default: None
    :param img_mask_path:   Path of the mask image, must be of the same size of input clip: Default: None
    :param mask_dilation:   Mask dilation for video and flow masking. Default: 8
    :param neighbor_length: Length of local neighboring frames. Low values decrease the memory usage.
                            High values could help to improve the quality on fast moving sequences.
                            Default: 10
    :param ref_stride:      Stride of global reference frames. High values will allow to
                            reduce the memory usage and increase the inference speed, but could
                            affect the inference quality. Default: 10
    :param raft_iter:       Iterations for RAFT inference. Low values will increase the inference
                            speed but could affect the output quality. Default: 20
    :param mask_region:     Allow to restrict the region of the mask, format: (width, height, left, top).
                            The region must be big enough to allow the inference. Default: None
    :param weights_dir:     Path string of location of model weights.
    :param sc_threshold:    If > 0 represent the scene change threshold used to generate the reference frames for
                            ProPainter, range [0,1]. Default = 0.1
    :param sc_min_freq:     Minimum number of frames that must elapse between two accepted scene changes.
                            If > 0, it is clamped to a minimum of 5 frames. Default is 0 (no spacing enforced).
    :param enable_fp16:     If True use fp16 (half precision) during inference. Default: fp16 (for RTX30 or above)
    :param device_index:    Device ordinal of the GPU (if = -1 CPU mode is enabled). Default: 0
    :param inference_mode:  Enable/Disable torch inference mode. Default: False
    """

Let me know if this RC2 is working on your side.

Dan

P.S.
This implementation uses internal mutable state to track the last accepted scene change frame.
As such, it assumes sequential, single-threaded frame access (e.g., during linear encoding).
It is not safe for random access, multi-threaded evaluation, or caching with reordering.

[Selur]

Busy atm. trying to improve the Vapoursynth syntax highlighter, but will try tomorrow and report back.

Cu Selur

[Selur]

Had a quick go at it.
1. Loading file with white spaces in the name works fine now.
2. sc_min_freq work too
strange thing is, now the clip seems also process fine when sc_min_freq = 0
Thanks!

Cu Selur

[Dan64]

Ok, I just released the new ProPainter version v1.2.2

Thanks,
Dan