Deoldify Vapoursynth filter

[Dan64]

Hello Selur,

  I do agree with you that moving inside vsdeoldify the grain/noise removal it will be possible to use a more "bold" approach.
  I opted for writing the following degrain function:

def vs_degrain(clip: vs.VideoNode = None, strength: int = 0) -> vs.VideoNode:
   
    if strength == 0:
        return clip
    else:
        import havsfunc
   
    match strength:
        case 1:      
          dstr = 1.0
        case 2:
          dstr = 1.5
        case 3:
          dstr = 2.5
        case 4:
          dstr = 3.5
        case _:
            raise vs.Error("ddeoldify: not supported strength value: " + mode)     
   
    clip = clip.resize.Bicubic(format=vs.YUV444PS, matrix_s="709", range_s="full")  
    clip = havsfunc.KNLMeansCL(clip=clip, d=1, a=2, s=4, h=dstr, device_type="gpu", device_id=0)
    clip = clip.resize.Bicubic(format=vs.RGB24, matrix_in_s="709", range_s="full", dither_type="error_diffusion")
   
    return clip

 
 I decided to use KNLMeansCL because it is fast and can remove both grain and noise. I simplified the input and to the functions ddeoldify_main() and ddeoldify() I added only 1 parameter, that control the strenght of grain/noise removal, called "degrain_strength" (if = 0, is disabled).

  I attached the new RC7.

Thanks,
Dan

[Selur]

Will look at it tomorrow.

[Selur]

Okay, took a look.
Unless I overlook something, this is different from what I suggested. (filtering for deoldify, but keeping the luma details)
=> not implementing this
This is the same as applying the denoising before deoldify which Hybrid already can do, but in Hybrid you can use any denoiser not just KNLMeansCL.
This does make sense when deoldify is used outside of Hybrid, but in Hybrid, adding this seems like a step back.

Cu Selur

[Dan64]

The implemented denoise/degrain will preserve the luma details.

The main code is the following:

def ddeoldify(
    clip: vs.VideoNode, method: int = 2, mweight: float = 0.5, deoldify_p: list = [0, 24, 1.0, 0.0], ddcolor_p: list = [1, 24, 1.0, 0.0, True], dotweak: bool = False, dotweak_p: list = [0.0, 1.0, 1.0, False, 0.2, 0.5, 1.5, 0.5], ddtweak: bool = False, ddtweak_p: list = [0.0, 1.0, 2.5, True, 0.3, 0.6, 1.5, 0.5, "300:360|0.8,0.1"],  degrain_strength: int = 0, cmc_tresh: float = 0.2, lmm_p: list = [0.2, 0.8, 1.0], alm_p: list = [0.8, 1.0, 0.15], cmb_sw: bool = False, device_index: int = 0, torch_dir: str = model_dir) -> vs.VideoNode:

...
...
chroma_resize = True  # always enabled
...
...
if chroma_resize:
        frame_size = min(max(ddcolor_rf, deoldify_rf) * 16, clip.width)     # frame size calculation for inference() 
        clip_orig = clip;
        clip = clip.resize.Spline64(width=frame_size, height=frame_size)
   
try:
    d_clip = vs_degrain(clip, degrain_strength)
except:
    vs.core.log_message(2, "Degrain filter: havsfunc.KNLMeansCL is not installed/loaded properly") 
    d_clip = clip 
   
clip = d_clip
...
...
if chroma_resize:
        return _clip_chroma_resize(clip_orig, clip_colored)
    else:
        return clip_colored

Since it was decided to remove the flag "chroma_resize", now this flag is always enabled.
This implies that no matter all the changes that are made to the luma by the filters, at the end the original luma is always restored.

Dan

P.S.
I discovered a bug in managing the exception on vs_degrain, I attached the correct version.

[Selur]

Ahh,..
chroma_resize_enabled is always true, which calls _clip_chroma_resize, which calls vs_recover_clip_luma
which is:

def vs_recover_clip_luma(orig: vs.VideoNode = None, clip: vs.VideoNode = None) -> vs.VideoNode:
    def copy_luma_frame(n, f):    
        img_orig = frame_to_image(f[0])
        img_clip = frame_to_image(f[1])        
        img_m = chroma_post_process(img_clip, img_orig)
        return image_to_frame(img_m, f[0].copy())                
    clip = clip.std.ModifyFrame(clips=[orig, clip], selector=copy_luma_frame)
    return clip

def chroma_post_process(img_m: Image, orig: Image) -> Image:
    img_np = np.asarray(img_m)
    orig_np = np.asarray(orig)
    img_yuv = cv2.cvtColor(img_np, cv2.COLOR_RGB2YUV)
    # copy the chroma parametrs "U", "V", of "img_m" in "orig"
    orig_yuv = cv2.cvtColor(orig_np, cv2.COLOR_RGB2YUV)
    orig_copy = np.copy(orig_yuv)
    orig_copy[:, :, 1:3] = img_yuv[:, :, 1:3]
    img_np_new = cv2.cvtColor(orig_copy, cv2.COLOR_YUV2RGB)
    return Image.fromarray(img_np_new)

which hopefully does the correct thing and is similar fast do Vapoursynth ShufflePlanes.

Cu Selur

[Selur]

Why are you using havsfunc?

clip = havsfunc.KNLMeansCL(clip=clip, d=1, a=2, s=4, h=dstr, device_type="gpu", device_id=0)

better use KNLMeansCL directly.

clip = core.knlm.KNLMeansCL(clip=clip, d=1, a=2, s=4, h=dstr, channels='Y', device_type="gpu", device_id=0)

No need to add another dependency.

https://github.com/Selur/VapoursynthScri...c.py#L5948
is only used to filter 'Y' and 'UV' separately to speed it up, but in your case there is no need to filter 'UV' at all.

Cu Selur

Ps.: uploaded a test version which does include KNLMeans, but not havsfunc.

[Dan64]

Thanks for the advice, I will update the code.

Dan

[Selur]

No, problem will release a new Hybrid tomorrow then, assuming you release a new DeOldify.

Cu Selur

[Dan64]

Ok, I will also release the new version tomorrow.

Thanks,
Dan

---

Ps.: uploaded a test version which does include KNLMeans, but not havsfunc.

I rewrote the "degrain" function as following

def vs_degrain(clip: vs.VideoNode = None, strength: int = 0, device_id: int = 0) -> vs.VideoNode:
   
    if strength == 0:
        return clip
   
    match strength:
        case 1:       
          dstr = 1.5
          dtmp = 1
        case 2:
          dstr = 2.5
          dtmp = 1
        case 3:
          dstr = 3.5
          dtmp = 1
        case 4:
          dstr = 5.5
          dtmp = 2
        case 5:
          dstr = 8.5
          dtemp = 2
        case _:
            raise vs.Error("ddeoldify: not supported strength value: " + strength)     
   
    clip = clip.resize.Bicubic(format=vs.YUV444PS, matrix_s="709", range_s="full")   
    clip = vs.core.knlm.KNLMeansCL(clip=clip, d=dtemp, a=2, s=4, h=dstr, channels='Y', device_type="gpu", device_id=device_id)
    clip = clip.resize.Bicubic(format=vs.RGB24, matrix_in_s="709", range_s="full", dither_type="error_diffusion")
   
    return clip

I prefer that in Hybrid the Degrain will display only integer values from 1 to 5. 
So that in the future I can decide to implement a different mapping and/or denoise/degrain function, without the need to change the input.

If you prefer you can call this parameter also "DegrainFactor".

Thanks

[Selur]

Uploaded a new test version.

Cu Selur