Improving CodeFormer videos

[Selur]

It is theoretically possible to extend "VsViewer" so that can be launched using the command line like "VSpipe" ?

if your rewrite most of it,... yes

vspipe should normally more stable than vsviewer.
One thing to look at is any anti virus software aside from windows defender,...

CUDA is backward compatible with the previous versions ...

are you sure? iirc CUDA 12 isn't always backward compatible (could be wrong there, but I think I read that somewhere when I came out, that it wasn't backward compatible due to aiming for newer cards only,...)

Cu Selur

[Dan64]

The question is quite simple.

Currently to encode a movie using a Vapoursynth script are available 2 tools:

1) VsPipe.exe
2) VsViewer.exe

If you know other tools please tell me which ones they are.

I'd like to try to understand how I can extend VsViewer, could you provide the steps necessary to compile the current version ?

Thanks,
Dan

[Dan64]

I discovered that StaxRip is distributed with a version of ffmpeg compiled with the option "--enable vapoursynth"

I copied this version in the folder "Hybrid\64bit\Vapoursynth" and then executed the following command

D:\Programs\Hybrid\64bit\Vapoursynth\ffmpeg -f vapoursynth -i CodeFormerTest6-001-BAD-1.vpy -c:v libx265 -c:a copy CodeFormerTest6-001-BAD-1.mp4

The movie was compiled successfully with no errors.

In this way it is possible to see also the messages printed by the filters.
In this case there were a lot of messages with text:

"No face detected. Try to increase upsample_num_times.:14.18 bitrate= 147.9kbits/s speed=0.654x"

Probably are all these messages that are creating problems to vspipe.

So, no need to change VsViewer.

It is just enough to add an option to use ffmpeg instead of vspipe. What do you think ?

Dan

---

I tried to use ffmpeg exactly as vspipe, using the following command

D:\Programs\Hybrid\64bit\Vapoursynth\ffmpeg -f vapoursynth -i CodeFormerTest6-001-BAD-1.vpy -vcodec rawvideo -strict -1 -f yuv4mpegpipe - | "D:\Programs\Hybrid\64bit\x265.exe" --y4m - -o "outputfile.265"

and I got the same problem, but this time with more details regarding the problem.

Output #0, yuv4mpegpipe, to 'pipe:':
  Metadata:
    encoder         : Lavf60.20.100
  Stream #0:0: Video: rawvideo (Y3[11][10] / 0xA0B3359), yuv420p10le(bt709/unknown/unknown, progressive), 720x392, q=2-31, 101504 kb/s, 23.98 fps, 23.98 tbn
      Metadata:
        encoder         : Lavc60.36.100 rawvideo
y4m  [info]: 720x392 fps 24000/1001 i420p10 unknown frame count
raw  [info]: output file: outputfile.265
x265 [info]: HEVC encoder version 3.5+115-88fd6d3ad
x265 [info]: build info [Windows][GCC 13.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-3 (Main tier)
x265 [info]: Thread pool created using 20 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 4 / wpp(7 rows)
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 3
x265 [info]: Keyframe min / max / scenecut / bias  : 23 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / off / on
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : CRF-28.0 / 0.60
x265 [info]: tools: rd=3 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
x265 [info]: tools: b-intra strong-intra-smoothing deblock sao
x265 [error]: y4m: frame header missing3kB time=00:00:12.30 bitrate=161858.5kbits/s speed=0.701x
x265 [info]: frame I:      2, Avg QP:25.15  kb/s: 2335.46
x265 [info]: frame P:     73, Avg QP:27.69  kb/s: 528.43
x265 [info]: frame B:    221, Avg QP:34.98  kb/s: 28.12
x265 [info]: Weighted P-Frames: Y:0.0% UV:0.0%

encoded 296 frames in 17.57s (16.85 fps), 167.10 kb/s, Avg QP:33.12
[vost#0:0/rawvideo @ 00000264b71ee700] Error submitting a packet to the muxer: Broken pipe
    Last message repeated 1 times
[out#0/yuv4mpegpipe @ 00000264b594d6c0] Error muxing a packet
[out#0/yuv4mpegpipe @ 00000264b594d6c0] Task finished with error code: -32 (Broken pipe)
[out#0/yuv4mpegpipe @ 00000264b594d6c0] Terminating thread with return code -32 (Broken pipe)
[out#0/yuv4mpegpipe @ 00000264b594d6c0] Error writing trailer: Broken pipe
[out#0/yuv4mpegpipe @ 00000264b594d6c0] Error closing file: Broken pipe
[out#0/yuv4mpegpipe @ 00000264b594d6c0] video:245582kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
frame=  297 fps= 17 q=-0.0 Lsize=  244757kB time=00:00:12.42 bitrate=161319.0kbits/s speed= 0.7x

It seems that the tons of messages generates using "dlib" have broken the pipe.

Avoiding the pipe will allow to fully encode the movie.

P.S.
In effect "dlib" is refusing to recognize a face when is not well visible. Vice-versa "retinaface" try to recognize the face in any case, but often the output has a lot of artifacts, try to encode the movie using "retinaface" and you will see the damage introduced by the wrong choice performed by "retinaface".

[Selur]

"No face detected. Try to increase upsample_num_times.:14.18 bitrate= 147.9kbits/s speed=0.654x"

Okay, what happens seems to be that your dlib version outputs stuff to std::out which is where the normal processed data is output and piped, thus is breaks the video data flow and breaks the pipe.
Seems like this is a problem of dlib. Maybe vs-codeformer can capture these outputs, but I see no way to stop this from breaking the pipes.

Cu Selur

[Dan64]

"No face detected. Try to increase upsample_num_times.:14.18 bitrate= 147.9kbits/s speed=0.654x"

Okay, what happens seems to be that your dlib version outputs stuff to std::out which is where the normal processed data is output and piped, thus is breaks the video data flow and breaks the pipe.

Is not a problem introduced in my version. Even the not enabled CUDA version is providing exactly the same message, the difference is that the CUDA version write a message every 0.4 secs while the CPU version every 20 secs.

I commented all the unnecessary print messages  (good only for development) in the filter vscodeformer.

I commented out the "print" in the following lines

->face_restoration_helper.py
Line 183:            print('No face detected. Try to increase upsample_num_times.')
Line 187:                print('Detect several faces and only keep the largest.')
->matlab_cp2tform.py
Line 282:    print('\n--->uv:')
Line 283:    print(uv)
Line 284:    print('\n--->xy:')
Line 285:    print(xy)
Line 289:    print('\n--->trans matrix:')
Line 290:    print(trans)
Line 292:    print('\n--->trans_inv matrix:')
Line 293:    print(trans_inv)
Line 295:    print('\n---> apply transform to uv')
Line 296:    print('\nxy_m = uv_augmented * trans')
Line 299:    print(xy_m)
Line 301:    print('\nxy_m = tformfwd(trans, uv)')
Line 303:    print(xy_m)
Line 305:    print('\n---> apply inverse transform to xy')
Line 306:    print('\nuv_m = xy_augmented * trans_inv')
Line 309:    print(uv_m)
Line 311:    print('\nuv_m = tformfwd(trans_inv, xy)')
Line 313:    print(uv_m)
Line 316:    print('\nuv_m = tforminv(trans, xy)')
Line 317:    print(uv_m)

 
and then cleared the cache.

Now the vspipe is working as expected. 

I will write to  HolyWu  to do the same on the version released on "github".

Dan 

P.S.
In any case it could be a big enhancement to introduce "ffmpeg" as Video encoder like "nvenc". 
In this case it will be possible to skip "vspipe" providing to Hybrid a more robust alternative in case of "unknown" problems with filters.

[Selur]

In any case it could be a big enhancement to introduce "ffmpeg" as Video encoder like "nvenc"

a. Hybrid already supports this on Linux
b. don't see any gain in it compared to NVEncC

In this case it will be possible to skip "vspipe" providing to Hybrid a more robust alternative in case of "unknown" problems with filters.

In my eyes, ffmpegs Vapoursynth support is not more stable than vspipe.
=> not planning to support directly loading the a vapoursynth script through ffmpeg

Cu Selur

[Dan64]

I opened the issue to HolyWu: https://github.com/HolyWu/vs-codeformer/issues/7

[Selur]

I wonder whether vs-codeformer code could simply capture those outputs instead of having to modify dlib.

[Dan64]

I wonder whether vs-codeformer code could simply capture those outputs instead of having to modify dlib.

The proposed changes are in the vs-codeformer package. I not changed "dlib", so HolyWu should be able to fix this issue.

P.S.
I attached an archive with the patched filter code.

[Selur]

Ah, okay, thought they were in the dlib wheel or package.

Cu Selur