Issues found during messing around with Hybrid

Issues found during messing around with Hybrid - Printable Version

+- Selur's Little Message Board (https://forum.selur.net)
+-- Forum: Hybrid - Support (https://forum.selur.net/forum-1.html)
+--- Forum: Problems & Questions (https://forum.selur.net/forum-3.html)
+--- Thread: Issues found during messing around with Hybrid (/thread-1770.html)

Issues found during messing around with Hybrid - admpodg - 11.03.2021

Hello,
During testing and trying different option with VHS captures I found some specific issues:

CCD (Camcorder Color Denoise) can not be multithreaded (there is no such checkbox and I see in script that this is set by default to 0) - when I was using this singlecore version, I have got only 17fps, while in VirtualDub I can manage 60fps of processing - it would be nice to turn multithreading ON in Hybrid for that plugin (or just adding ability to set MT parameter to 1 )
aomenc coder does not encode when I choose lossless mode (logs attached)

RE: Issues found during messing around with Hybrid - Selur - 11.03.2021

About aomenc:
Error message is:

Quote:Error: Tried to set control 40 = 1
Failed to control codec: Invalid parameter
Only --aq_mode=0 can be used with --lossless=1.

-> will adjust Hybrid to respect this, so until a new version is release, simply setting, "aomenc->Base->Miscellaneous->Adaptive qunatization" to 'none' should work.

About CCD:
using:

ClearAutoloadDirs()

SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)

LoadPlugin("I:\Hybrid\32bit\PLUGIN~1\VDubFilter.dll")

LoadVirtualDubPlugin("I:\Hybrid\32bit\avisynthPlugins\ccd_32bit.vdf","VD_CCD",0)

LoadCPlugin("I:\Hybrid\32bit\AVISYN~1\ffms2.dll")

# loading source: F:\TestClips&Co\files\test.avi

#  color sampling YV12@8, matrix: bt601, scantyp: progressive, luminance scale: limited

FFVideoSource("F:\TESTCL~1\files\test.avi",cachefile="E:\Temp\avi_078c37f69bb356e7b5fa040c71584c40_853323747_1_0.ffindex",fpsnum=25,colorspace="YV12")

# current resolution: 640x352

# filtering

# camcorder color denoise

ConvertToRGB32(interlaced=false, matrix="Rec601")

VD_CCD(30,0)

# adjust color to YV12 (color matrix: Rec601)

ConvertToYV12(matrix="Rec601")

PreFetch(8)

# setting output fps to 25.000fps

AssumeFPS(25,1)

#  output: color sampling YV12@8, matrix: bt601, scantyp: progressive, luminance scale: limited

return last

gives me:

Frames processed:                   429 (0 - 428)

FPS (min | max | average):          106.9 | 4326 | 373.3

Process memory usage (max):         92 MiB

Thread count:                       45

CPU usage (average):                34.1%

Time (elapsed):                     00:00:01.149

disabling MT: "Filtering->Avisynth->Miscellaneous->Disable MT"
script looks like:

ClearAutoloadDirs()

LoadPlugin("I:\Hybrid\32bit\PLUGIN~1\VDubFilter.dll")

LoadVirtualDubPlugin("I:\Hybrid\32bit\avisynthPlugins\ccd_32bit.vdf","VD_CCD",0)

LoadCPlugin("I:\Hybrid\32bit\AVISYN~1\ffms2.dll")

# loading source: F:\TestClips&Co\files\test.avi

#  color sampling YV12@8, matrix: bt601, scantyp: progressive, luminance scale: limited

FFVideoSource("F:\TESTCL~1\files\test.avi",cachefile="E:\Temp\avi_078c37f69bb356e7b5fa040c71584c40_853323747_1_0.ffindex",fpsnum=25,colorspace="YV12")

# current resolution: 640x352

# filtering

# camcorder color denoise

ConvertToRGB32(interlaced=false, matrix="Rec601")

VD_CCD(30,1)

# adjust color to YV12 (color matrix: Rec601)

ConvertToYV12(matrix="Rec601")

# setting output fps to 25.000fps

AssumeFPS(25,1)

#  output: color sampling YV12@8, matrix: bt601, scantyp: progressive, luminance scale: limited

return last

and gives me:

Frames processed:                   429 (0 - 428)

FPS (min | max | average):          137.6 | 203.1 | 192.9

Process memory usage (max):         43 MiB

Thread count:                       36

CPU usage (average):                12.9%

Time (elapsed):                     00:00:02.224

so for me that is slower,...
My gues is that using native VirtualDub filters in Avisynth simply is slower,...

Cu Selur

RE: Issues found during messing around with Hybrid - admpodg - 11.03.2021

Thanks for quick reply! Smile

I will check again aomenc - thanks for the tips, will be looking more carefully into the logs.

About CCD - I assume that this is done in that way to avoid some risky behaviour? While Avisynth will be in MT mode and also CCD, there could be multiple instances of CCD, which additionally will use 4 threads per instance (as per CCD gui in VD), am I right?

RE: Issues found during messing around with Hybrid - Selur - 11.03.2021

Haven't used VirtualDub for over 10 years, but assuming the 4 threads is something CCD has internally I would say: yes.
If it's a feature of VirtualDub then: no.
Atm. I supect it's not something the filter uses itself, but rather some function in VirtualDub.

Cu Selur

RE: Issues found during messing around with Hybrid - admpodg - 11.03.2021

It is just an option in GUI of CCD filter, which probably corresponds to second argument of CCD function in AVS script:
[Image: 3eGGsls.png]

With MT enabled VD renders a file twice as much faster than singlethreaded. I am just curious what it would happen if both Avisynth and CCD were in MT mode Wink

RE: Issues found during messing around with Hybrid - Selur - 11.03.2021

Quote:With MT enabled VD renders a file twice as much faster than singlethreaded.

Hmm,...
with Avisynth's MT disabled and using:

VD_CCD(30,1) I get:

Frames processed: 429 (0 - 428) FPS (min | max | average): 142.4 | 212.8 | 204.1 Process memory usage (max): 43 MiB Thread count: 36 CPU usage (average): 13.7% Time (elapsed): 00:00:02.102
VD_CCD(30,0) I get:

Frames processed: 429 (0 - 428) FPS (min | max | average): 20.54 | 85.68 | 66.00 Process memory usage (max): 43 MiB Thread count: 36 CPU usage (average): 6.0% Time (elapsed): 00:00:06.500

with Avisynth's MT enabled and using:

VD_CCD(30,1) I get:

Frames processed: 429 (0 - 428) FPS (min | max | average): 134.6 | 1131 | 566.8 Process memory usage (max): 90 MiB Thread count: 44 CPU usage (average): 37.1% Time (elapsed): 00:00:00.757
VD_CCD(30,0) I get:

Frames processed: 429 (0 - 428) FPS (min | max | average): 165.1 | 1661 | 503.0 Process memory usage (max): 90 MiB Thread count: 44 CPU usage (average): 26.4% Time (elapsed): 00:00:00.853

So it, does make a difference,...
-> I'll add an 'MT' option, which when enabled will use VD_CCD(X,1) and when disabled VD_CCD(X,0).
Will send you a link to a dev version in ~30min.

Cu Selur

RE: Issues found during messing around with Hybrid - Selur - 11.03.2021

Send link, also let me know if it works properly.
Iirc I set '0' by hardcoded, since using '1' did cause problems/crashes when 'Avisynth MT' instead of the current 'Avisynth++ MT' was used.
So I'm still a bit vary about it, so let me know in cause it still causes crashes or if it works fine with Avisynth++ MT. Smile

Cu Selur

RE: Issues found during messing around with Hybrid - admpodg - 11.03.2021

Many thanks for dev binary Smile

On my side it looks more interesting:
-test setup: Ryzen 5 3600, 16GB RAM;
-source video: 1 minute 50fps deinterlaced VHS capture encoded in FFV1
-CCD filter and encoding again to FFV1

...and:

AS MT ON, CCD MT ON - 101 fps
AS MT ON, CCD MT OFF - 101 fps
AS MT OFF, CCD MT OFF - 35-37fps
AS MT OFF, CCD MT ON - 71fps

The two last scores are similar to VD, with MT enabled Hybrid performs very well - to be honest I don't know why I had such poor results in the first post - I assume that deinterlacer was turned on - I will check it again on official release, but tommorow Smile

Only during adding new job I had error pop-up: "FilteringModel get->found no field addCustomScriptCNR2", which had to be clicked a couple of times, then it dissapeared

RE: Issues found during messing around with Hybrid - admpodg - 12.03.2021

I run tests again, this time I unistalled every version properly, so there is no popup error. I compared both version, I see now no difference in fps, on both version specific values are the same. Still no idea why I got results from first post:/ More interesting things, I checked encoding with NVenc to h264 - and in my case fps got bigger (100 vs 125) when I disabled MT in CCD (Avisynth in MT mode). So there is plenty of room for testing different configurations Smile

Anyway, I see big potential in Hybrid, I finally found a tool which can perform QTGMC much faster than other programs and big + for those filters I am still experimenting with. Keep up! Smile