Improving CodeFormer videos

[Dan64]

The new dev version is working now.
But I'm unable to use FeMaSR tensor-RT with CodeFormer.
If I use them separately (in different scripts) they are working, but together no.
Did you changed CUDA library recently ?

[Selur]

Not really, fi you lookt at Hybrid\64bit\Vapoursynth\torch_dependencies\bin you can see that those libraries are all rather old.
No clue where the problem is, might be some issue with FP16,.. will to some testing. Or some incompatibility between CodeFormer and TRT,...
as you can seee there also have been isues with FeMaSR and TRT,... https://github.com/HolyWu/vs-femasr/issues
=> I'll remove the trt options,..

Cu Selur

[Dan64]

  It is theoretically possible to extend "VsViewer" so that can be launched using the command line like "VSpipe" ?

if your rewrite most of it,... yes

Hello Selur,

  I applied some changes to vsViewer released in your github repo: https://github.com/Selur/vsViewer/tree/master

 The usage is the following
 
 

vsViewer --nogui [optional] --y4m [optional] --i [python script]  --enc [encoder path] --cmd [encode command]

where:

--nogui   : if present the encoding gui will not be visible
--y4m     : if present will be provided the y4m header
--i       : input vapoursynth script
--enc     : encoder program
--cmd     : encoder command

Examples:

1) Encoding with encoding Gui and with y4m header (*)
vsViewer.exe --y4m --i "VideoScript.vpy"  --enc "x265.exe" --cmd "- --y4m  --crf 18 --preset fast --output-depth 10 -o \"{sd}\{sn}_x265_gui.hevc\""

2) Encoding with no-Gui and with y4m header (*)
vsViewer.exe --gui --y4m --i "VideoScript.vpy"  --enc "x265.exe" --cmd "- --y4m  --crf 18 --preset fast --output-depth 10 -o \"{sd}\{sn}_x265_gui.hevc\""


(*) for script and encoder pass the full path not only the name as in this example

 
  I  like the Gui enabled version please try it, in case of no-Gui I was unable to write any message on Console (I have only a limited knowledge of Qt).

  I attached an archive with only the changed files and with a build in debug mode of vsViewer.

  I tried to don't apply too many changes, more important this version is compatible with the version release with Hybrid.

  I hope that this modded version can be useful (you can abandon "vspipe" and all the encoding will be under your control) 

Dan

[Selur]

Likes I wrote before I don't really see any meaningful benefit from using vsViewer instead of vspipe.
vsViewer was a mod of https://github.com/YomikoR/VapourSynth-Editor which mainly added IPC functionality. I later modified the source to be Qt6 compatible.
That vsViewer even can encode stuff is a remnant from the original.
Looking at your changes:

bool initialized = m_pJob->initialize();

(in MainWindowlotBatchEncode; initialized isn't used anywhere,...)[/code]
is something missing?

Cu Selur

[Dan64]

m_pJob->initialize()

is necessary to initialize the job.
I should check the return and eventually manage the exception.
But this part has not been implemented.
Aldo I don't found a way to write to console when no-Gui mode. 

[Selur]

There isn't a (good) way unless your program is compiled for command line only or it does open a terminal (on Linux&Co this is Windows is not shown, on Windows it is).
(add "CONFIG += console" to the .pro file)
Usually this isn't really a problem, since writing to std::out from a gui, doesn't make much sense.
One point of having a gui is not to have to deal with std::out.


Cu Selur

[Dan64]

I think that the best use is with the IPC comunication already implanted in Hybrid (*).
In this way you can capture all the messages and monitor the progress with more precision.
Regarding "vspipe" there are several post in the forum regarding the fact that a given script works in "vsViewer" but fails in the encoder.

Dan

(*) If necessary I can apply the extension to the code for you.

[Selur]

Regarding "vspipe" there are several post in the forum regarding the fact that a given script works in "vsViewer" but fails in the encoder.

That usually happens if:
a. the script in vsViewer is different
b. if using an encoder eats more system resouces and causes the system to crash
c. some anti virus software interferes with the pipe (same would happen with vsviewer)
d. user changed components that write to std::out.
usually if preview work encoding works too unless I messed somewhere up.

Cu Selur

[Dan64]

Not really, fi you lookt at Hybrid\64bit\Vapoursynth\torch_dependencies\bin you can see that those libraries are all rather old.
No clue where the problem is, might be some issue with FP16,.. will to some testing. Or some incompatibility between CodeFormer and TRT,...
as you can seee there also have been isues with FeMaSR and TRT,... https://github.com/HolyWu/vs-femasr/issues
=> I'll remove the trt options,..

Cu Selur

Hello Selur,

  I installed the last version 2024.02.24.1, but the FeMaSR is not working on my PC.
  I got the error message: Python exception: "slow_conv2d_cpu" not implemented for 'Half'

  I performed some tests and I think that should be better add support to RealESRGAN 2Xplus/4Xplus and realesr-general

  You can find a comparison here: https://imgsli.com/MjQyNDU0
  To simplify the list, tensor-rt could be added as an option.

Dan

P.S.
I attached the log

[Selur]

File "E:\VideoTest\Deoldify\tempPreviewVapoursynthFile17_09_44_253.vpy", line 54, in
clip = ddcolor(clip=clip, model=0)

sounds like the problem is with ddcolor not CodeFormer.
My guess it that your are running out of VRAM while throwing feeding ddcolor with 2400x1632 content.
Otherwis the script:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
import ctypes
import os
import site
core = vs.core
# Adding torch dependencies to PATH
path = site.getsitepackages()[0]+'/torch_dependencies/bin/'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]
# Loading Plugins
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="D:/Programs/Hybrid/64bit/vsfilters/Support/libimwri.dll")
# source: 'E:/VideoTest/Deoldify/TestBW0.jpg'
# current color space: YUV420P8, bit depth: 8, resolution: 800x544, fps: 25, scanorder: progressive, yuv luminance scale: full, matrix: 709
# Loading E:\VideoTest\Deoldify\TestBW0.jpg using vsImageReader
clip = core.imwri.Read(["E:/VideoTest/Deoldify/TestBW0.jpg"])
clip = core.std.Loop(clip=clip, times=100)
# Input color space is assumed to be RGB24
frame = clip.get_frame(0)
# Setting color transfer (170), if it is not set.
if '_Transfer' not in frame.props or not frame.props['_Transfer']:
  clip = core.std.SetFrameProps(clip, _Transfer=6)
# Setting color primaries info (to 470), if it is not set.
if '_Primaries' not in frame.props or not frame.props['_Primaries']:
  clip = core.std.SetFrameProps(clip, _Primaries=5)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
# Blind Face Restoration using CodeFormer
# Using background resizer FeMaSR(2x)
clipBG = clip
from vsfemasr import femasr as FeMaSR
# adjusting color space from RGB24 to RGBH for vsFeMaSR
clipBG = core.resize.Bicubic(clip=clipBG, format=vs.RGBH, range_s="full")
# Step 1: current: 800x544, target: 1600x1088
# resizing using FeMaSR
clipBG = FeMaSR(clip=clipBG, device_index=0, num_streams=3) # 1600x1088
# resizing 1600x1088 to 1600x1088
# adjusting resizing
clipBG = core.resize.Bicubic(clip=clipBG, format=vs.RGBS, range_s="full")
clipBG = core.fmtc.resample(clip=clipBG, w=1600, h=1088, kernel="spline64", interlaced=False, interlacedd=False)
clipBG = core.resize.Bicubic(clip=clipBG, format=vs.RGB24, range_s="full", dither_type="error_diffusion")
from vscodeformer import codeformer as CodeFormer
clip = CodeFormer(clip=clip, upscale=2, weight=0.000, num_streams=3, bg_clip=clipBG) # 1600x1088
# Resizing using fmtconv - spline36
clip = core.fmtc.resample(clip=clip, kernel="spline36", w=2400, h=1632, interlaced=False, interlacedd=False) # resolution 2400x1632 before RGB24 after RGB48
# adjusting color space from RGB48 to RGBH for vsDDColor
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, range_s="full")
# adding colors using DDColor
from vsddcolor import ddcolor
clip = ddcolor(clip=clip, model=0)
# adjusting output color from: RGBH to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="709", range_s="full", dither_type="error_diffusion")
# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

looks fine to me.

CodeFormer+FeMaSR + ddcolor works here, untill I use a too high resolution.
I leave the options as they are for the time being, since as it is now it works and lowering the ComboBox option count and at the same time having to add an additional CheckBox and model entry doesn't really simplify things.

Cu Selur