Issues found during messing around with Hybrid

[admpodg]

Hello,
During testing and trying different option with VHS captures I found some specific issues:

1. CCD (Camcorder Color Denoise) can not be multithreaded (there is no such checkbox and I see in script that this is set by default to 0) - when I was using this singlecore version, I have got only 17fps, while in VirtualDub I can manage 60fps of processing - it would be nice to turn multithreading ON in Hybrid  for that plugin (or just adding ability to set MT parameter to 1 )
   
2. aomenc coder does not encode when I choose lossless mode (logs attached)
   

[Selur]

About aomenc:
Error message is:

Error: Tried to set control 40 = 1
Failed to control codec: Invalid parameter
Only --aq_mode=0 can be used with --lossless=1.

-> will adjust Hybrid to respect this, so until a new version is release, simply setting, "aomenc->Base->Miscellaneous->Adaptive qunatization" to 'none' should work.

About CCD:
using:

ClearAutoloadDirs()
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
LoadPlugin("I:\Hybrid\32bit\PLUGIN~1\VDubFilter.dll")
LoadVirtualDubPlugin("I:\Hybrid\32bit\avisynthPlugins\ccd_32bit.vdf","VD_CCD",0)
LoadCPlugin("I:\Hybrid\32bit\AVISYN~1\ffms2.dll")
# loading source: F:\TestClips&Co\files\test.avi
#  color sampling YV12@8, matrix: bt601, scantyp: progressive, luminance scale: limited
FFVideoSource("F:\TESTCL~1\files\test.avi",cachefile="E:\Temp\avi_078c37f69bb356e7b5fa040c71584c40_853323747_1_0.ffindex",fpsnum=25,colorspace="YV12")
# current resolution: 640x352
# filtering
# camcorder color denoise
ConvertToRGB32(interlaced=false, matrix="Rec601")
VD_CCD(30,0)
# adjust color to YV12 (color matrix: Rec601)
ConvertToYV12(matrix="Rec601")
PreFetch(8)
# setting output fps to 25.000fps
AssumeFPS(25,1)
#  output: color sampling YV12@8, matrix: bt601, scantyp: progressive, luminance scale: limited
return last

gives me:

Frames processed:                   429 (0 - 428)
FPS (min | max | average):          106.9 | 4326 | 373.3
Process memory usage (max):         92 MiB
Thread count:                       45
CPU usage (average):                34.1%

Time (elapsed):                     00:00:01.149

disabling MT: "Filtering->Avisynth->Miscellaneous->Disable MT"
script looks like:

ClearAutoloadDirs()
LoadPlugin("I:\Hybrid\32bit\PLUGIN~1\VDubFilter.dll")
LoadVirtualDubPlugin("I:\Hybrid\32bit\avisynthPlugins\ccd_32bit.vdf","VD_CCD",0)
LoadCPlugin("I:\Hybrid\32bit\AVISYN~1\ffms2.dll")
# loading source: F:\TestClips&Co\files\test.avi
#  color sampling YV12@8, matrix: bt601, scantyp: progressive, luminance scale: limited
FFVideoSource("F:\TESTCL~1\files\test.avi",cachefile="E:\Temp\avi_078c37f69bb356e7b5fa040c71584c40_853323747_1_0.ffindex",fpsnum=25,colorspace="YV12")
# current resolution: 640x352
# filtering
# camcorder color denoise
ConvertToRGB32(interlaced=false, matrix="Rec601")
VD_CCD(30,1)
# adjust color to YV12 (color matrix: Rec601)
ConvertToYV12(matrix="Rec601")
# setting output fps to 25.000fps
AssumeFPS(25,1)
#  output: color sampling YV12@8, matrix: bt601, scantyp: progressive, luminance scale: limited
return last

and gives me:

Frames processed:                   429 (0 - 428)
FPS (min | max | average):          137.6 | 203.1 | 192.9
Process memory usage (max):         43 MiB
Thread count:                       36
CPU usage (average):                12.9%

Time (elapsed):                     00:00:02.224

so for me that is slower,...
My gues is that using native VirtualDub filters in Avisynth simply is slower,...


Cu Selur

[admpodg]

Thanks for quick reply!

I will check again aomenc - thanks for the tips, will be looking more carefully into the logs.

About CCD - I assume that this is done in that way to avoid some risky behaviour? While Avisynth will be in MT mode and also CCD, there could be multiple instances of CCD, which additionally will use 4 threads per instance (as per CCD gui in VD), am I right?

[Selur]

Haven't used VirtualDub for over 10 years, but assuming the 4 threads is something CCD has internally I would say: yes.
If it's a feature of VirtualDub then: no.
Atm. I supect it's not something the filter uses itself, but rather some function in VirtualDub.

Cu Selur

[admpodg]

It is just an option in GUI of CCD filter, which probably corresponds to second argument of CCD function in AVS script: 


With MT enabled VD renders a file twice as much faster than singlethreaded. I am just curious what it would happen if both Avisynth and CCD were in MT mode

[Selur]

With MT enabled VD renders a file twice as much faster than singlethreaded.

Hmm,...
with Avisynth's MT disabled and using:

• VD_CCD(30,1) I get:
  
  Frames processed:                   429 (0 - 428)
FPS (min | max | average):          142.4 | 212.8 | 204.1
Process memory usage (max):         43 MiB
Thread count:                       36
CPU usage (average):                13.7%

Time (elapsed):                     00:00:02.102
• VD_CCD(30,0) I get:
  
  Frames processed:                   429 (0 - 428)
FPS (min | max | average):          20.54 | 85.68 | 66.00
Process memory usage (max):         43 MiB
Thread count:                       36
CPU usage (average):                6.0%

Time (elapsed):                     00:00:06.500

with Avisynth's MT enabled and using:

• VD_CCD(30,1) I get:
  
  Frames processed:                   429 (0 - 428)
FPS (min | max | average):          134.6 | 1131 | 566.8
Process memory usage (max):         90 MiB
Thread count:                       44
CPU usage (average):                37.1%

Time (elapsed):                     00:00:00.757
• VD_CCD(30,0) I get:
  
  Frames processed:                   429 (0 - 428)
FPS (min | max | average):          165.1 | 1661 | 503.0
Process memory usage (max):         90 MiB
Thread count:                       44
CPU usage (average):                26.4%

Time (elapsed):                     00:00:00.853

So it, does make a difference,...
-> I'll add an 'MT' option, which when enabled will use VD_CCD(X,1) and when disabled VD_CCD(X,0).
Will send you a link to a dev version in ~30min.

Cu Selur

[Selur]

Send link, also let me know if it works properly.
Iirc I set '0' by hardcoded, since using '1' did cause problems/crashes when 'Avisynth MT' instead of the current 'Avisynth++ MT' was used.
So I'm still a bit vary about it, so let me know in cause it still causes crashes or if it works fine with Avisynth++ MT.

Cu Selur

[admpodg]

Many thanks for dev binary
On my side it looks more interesting:
-test setup: Ryzen 5 3600, 16GB RAM; 
-source video: 1 minute 50fps deinterlaced VHS capture encoded in FFV1 
-CCD filter and encoding again to FFV1

...and:

AS MT ON, CCD MT ON   - 101 fps
AS MT ON, CCD MT OFF  - 101 fps
AS MT OFF, CCD MT OFF - 35-37fps
AS MT OFF, CCD MT ON  - 71fps 

The two last scores are similar to VD, with MT enabled Hybrid performs very well - to be honest I don't know why I had such poor results in the first post - I assume that deinterlacer was turned on - I will check it again on official release, but tommorow

Only during adding new job I had error pop-up: "FilteringModel get->found no field addCustomScriptCNR2",  which had to be clicked a couple of times, then it dissapeared

[admpodg]

I run tests again, this time I unistalled every version properly, so there is no popup error. I compared both version, I see now no difference in fps, on both version specific values are the same. Still no idea why I got results from first post:/ More interesting things, I checked encoding with NVenc to h264 - and in my case fps got bigger (100 vs 125) when I disabled MT in CCD (Avisynth in MT mode). So there is plenty of room for testing different configurations

Anyway, I see big potential in Hybrid, I finally found a tool which can perform QTGMC much faster than other programs and big + for those filters I am still experimenting with. Keep up!