Improving CodeFormer videos

[Dan64]

I read your document, but it is not clear how a filter can be added more times.
When I check the box "Use Filter Queue" the display change and shows the filters in use.
On the right there the symbol "-" to delete a row, but it is missing "+" to add a row, how I can add the same filter more times ?

[Selur]

After switching to 'Filter Queue' go to the filter and you will see a '+' and a '-' which allows you to add/remove a filter with a specific setting from the filter queue.
If you go do the filter queue you can also change the order of filters.

[Dan64]

Hello Selur,

  Usually CodeFormer is used with a Resizer.
  In your example I noted that you used SwinIR, which is a fast and good resizer, but RealESRGAN is better.
  In Hybrid it is possible to apply the RealESRGAN resizer and the CodeFormer.
  But CodeFormer allows to use the RealESRGAN resizer directly in the filter.
  This approach is not supported (are supported only simple resizer like Lanczos).
   I compared the result obtained using RealESRGAN  directly in the filter and the output image is better
   https://imgsli.com/MjM4NTgx 
 
   I used the following script

# Imports
import vapoursynth as vs
# getting Vapoursynth core
import ctypes
import os
import site
core = vs.core
# Adding torch dependencies to PATH
path = site.getsitepackages()[0]+'/torch_dependencies/bin/'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]
# Loading Plugins
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# source: 'E:\VideoTest\CodeFormer\Selur\suzie_lossless.mp4'
# current color space: YUV444P8, bit depth: 8, resolution: 176x144, fps: 29.97, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading E:\VideoTest\CodeFormer\Selur\suzie_lossless.mp4 using LWLibavSource
clip = core.lsmas.LWLibavSource(source="E:/VideoTest/CodeFormer/Selur/suzie_lossless.mp4", format="YUV444P8", stream_index=0, cache=0, fpsnum=30000, fpsden=1001, prefer_hw=0)
frame = clip.get_frame(0)
# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# Setting color transfer (to 470bg), if it is not set.
if '_Transfer' not in frame.props or not frame.props['_Transfer']:
  clip = core.std.SetFrameProps(clip, _Transfer=5)
# Setting color primaries info (to 5), if it is not set.
if '_Primaries' not in frame.props or not frame.props['_Primaries']:
  clip = core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 29.97
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
from vsrealesrgan import realesrgan as RealESRGAN
# adjusting color space from YUV444P8 to RGBH for vsRealESRGAN
clipBG = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_s="limited")
# resizing using RealESRGAN
clipBG = RealESRGAN(clip=clipBG, model=2, device_index=0, trt=True, trt_cache_path=r"E:\VideoTest\CodeFormer\Selur") # 704x576
# changing range from limited to full range
clipBG = core.resize.Bicubic(clipBG, range_in_s="limited", range_s="full")
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# adjusting color space from RGBH to RGB24 for vsCodeFormer
clipBG = core.resize.Bicubic(clip=clipBG, format=vs.RGB24, range_s="full", dither_type="error_diffusion")
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, range_s="full", dither_type="error_diffusion")
# Blind Face Restoration using CodeFormer
#clipBG = core.resize.Lanczos(clip, width=704, height=576) # 704x576
from vscodeformer import codeformer as CodeFormer
clip = CodeFormer(clip=clip, upscale=4, weight=1.000, only_center_face=False, num_streams=4, bg_clip=clipBG) # 704x576
# changing range from full to limited range
clip = core.resize.Bicubic(clip, range_in_s="full", range_s="limited")
# adjusting output color from: RGB24 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_s="limited")
# adjusted resizing to achieve PAR 1:1 for preview
# resizing for preview to square pixel to 770x576
clip = core.resize.Bicubic(clip=clip, width=770, height=576)
# set output frame rate to 29.97fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Output
clip.set_output()

   It is possible to add this type of resize in Hybrid ?

Thanks
Dan

P.S.
Given the special combination of Resizer and CodeFormer, could be better add this capability in the Resizer page of RealESRGAN, because after CodeFormer the video is upscaled

[Selur]

Yes, Hybrid could be extended to support additional resizers for the background upscaling.
Main problem is how to properly integrate this in the gui, since there are tons of resizers with tons of options.

Cu Selur

Ps.: just noticed the 'NNEDI3' as Upscaler is selectable, but I never implemented support for it. :/

[Dan64]

Hello Selur,

  I will left CodeFormer as is (eventually you can remove NNEDI3 since is available also in the Resizer Page).

  My suggestion is to extend the Resizer Page by adding a check box similar to "Stepped Resizer" called something like "CodeFormer Resizer"
  Where it will be possible to select the CodeFormer options: "Scale", "Weight", "Streams", "Only center" (the "dlib" detector is so slow that is not usable).
  In this way you can reuse the code already in place for the Resizer.
  The only difference is that the upscale selected in CodeFormer will force the Resizer output because they must match.  

Thanks,
Dan

[Selur]

Not a chance.
I will not add additional suff there.
Beeing able to combine Stepped Resize and Weighting is already a pain to code and debug.
If I add additional Resizers to CodeFormer I will probably add a few option without any additional configuration.

[Selur]

Send you a link to a dev version for testing.

Cu Selur

[Dan64]

Hello Selur,

  I tested "RIFE(general)" and is working as expected (good implementation).
  I was unable to have "RIFE(trt)" working.
  If I perform the resize using RealESRGAN with TRT enabled, the script is working, while the CodeFormer generated version not.

  I attached an archive with the 2 scripts generated and the log:

•      CodeFormerTest5_ESR_trt.vpy: is working
  
•      CodeFormerTest5_RIFE_trt.vpy: is not working
  

  I was unable to find a significant difference, maybe I'm too tired...

Thanks,
Dan

[Selur]

DOH, typo + missing argument.
RIFE should be ESRGAN
-> will fix

[Selur]

Send you a new link.