[HELP] Random FFmpeg crashes when using Hybrid / Avisynth

[The_Tinkerer]

I don't expect anyone to be able to figure this out due to the highly idiosyncratic and esoteric nature of the problem. But, I have wasted days on this now and I am tearing my hair out trying everything to troubleshoot this, to no avail.

The problem is, when I try to run jobs using Hybrid with Avisynth, I get flaky and unstable behavior. Sometimes, FFmpeg randomly crashes at random points in the job. Sometimes, all CPU activity stalls with no crash, but the job is now stuck. I can usually reset and restart the job and it will complete the second or third time.

Log Name:      Application
Source:        Application Error
Date:          4/18/2019 10:54:56 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      WIN-XXXXXXXXXXX
Description:
Faulting application name: ffmpeg.exe, version: 0.0.0.0, time stamp: 0x00000000
Faulting module name: KERNELBASE.dll, version: 6.1.7601.24408, time stamp: 0x5c92f101
Exception code: 0xc0000005
Fault offset: 0x0000c5af
Faulting process id: 0x13b4
Faulting application start time: 0x01d4f658174d46a1
Faulting application path: C:\PROGRA~1\Hybrid\32bit\ffmpeg.exe
Faulting module path: C:\Windows\syswow64\KERNELBASE.dll
Report Id: 82d22dc1-624e-11e9-b816-0050560b8935
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2019-04-19T02:54:56.000000000Z" />
    <EventRecordID>249111</EventRecordID>
    <Channel>Application</Channel>
    <Computer>WIN-XXXXXXXXXXX</Computer>
    <Security />
  </System>
  <EventData>
    <Data>ffmpeg.exe</Data>
    <Data>0.0.0.0</Data>
    <Data>00000000</Data>
    <Data>KERNELBASE.dll</Data>
    <Data>6.1.7601.24408</Data>
    <Data>5c92f101</Data>
    <Data>c0000005</Data>
    <Data>0000c5af</Data>
    <Data>13b4</Data>
    <Data>01d4f658174d46a1</Data>
    <Data>C:\PROGRA~1\Hybrid\32bit\ffmpeg.exe</Data>
    <Data>C:\Windows\syswow64\KERNELBASE.dll</Data>
    <Data>82d22dc1-624e-11e9-b816-0050560b8935</Data>
  </EventData>
</Event>

I now have a job that I am unable to complete without a crash or stall.

I have to set the number of MT threads to 12 in the Avisynth Misc tab or the job will run WAY too slow. All 6 cores are used, but the CPU utilization is never more than 20% and the performance is "spiky" in Task Manager, meaning there are only short "bursts" or spikes of activity in between no activity on all cores.

In addition to that, sometimes when starting Hybrid I get random crashes; all I can do is close and restart. It usually runs on restart. Sometimes I will get two crashes in a row until restarting successfully.

Log Name:      Application
Source:        Application Error
Date:          4/18/2019 11:36:10 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      WIN-XXXXXXXXXXX
Description:
Faulting application name: Hybrid.exe, version: 2019.3.13.0, time stamp: 0x5c89658b
Faulting module name: Qt5Gui.dll, version: 5.12.0.0, time stamp: 0x5bee693c
Exception code: 0xc00000fd
Fault offset: 0x0000000000109b7c
Faulting process id: 0x764
Faulting application start time: 0x01d4f66106a85341
Faulting application path: C:\Program Files\Hybrid\Hybrid.exe
Faulting module path: C:\Program Files\Hybrid\Qt5Gui.dll
Report Id: 457021c1-6254-11e9-b816-0050560b8935
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2019-04-19T03:36:10.000000000Z" />
    <EventRecordID>249116</EventRecordID>
    <Channel>Application</Channel>
    <Computer>WIN-XXXXXXXXXXX</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Hybrid.exe</Data>
    <Data>2019.3.13.0</Data>
    <Data>5c89658b</Data>
    <Data>Qt5Gui.dll</Data>
    <Data>5.12.0.0</Data>
    <Data>5bee693c</Data>
    <Data>c00000fd</Data>
    <Data>0000000000109b7c</Data>
    <Data>764</Data>
    <Data>01d4f66106a85341</Data>
    <Data>C:\Program Files\Hybrid\Hybrid.exe</Data>
    <Data>C:\Program Files\Hybrid\Qt5Gui.dll</Data>
    <Data>457021c1-6254-11e9-b816-0050560b8935</Data>
  </EventData>
</Event>

I seem to be able to run Vapoursynth without issue.

The reason I am using Avisynth is I am using amDCT Deblocker, which is not available in Vapoursynth. I suspect that this filter is what is causing most of the instability, but when it works it gives me the best results for what it does.

Do you have ANY idea on how to go about troubleshooting this?

I did try replacing C:\Program Files\Hybrid\32bit\ffmpeg.exe with ffmpeg.exe x86 version 4.1.1 (the latest release build), but it had no effect.


CPU-Z:

CPUZ.txt in Logs.zip


I have 5 logs. The only thing different between the logs are settings on the Avisynth Misc tab. Red boxes show what is changed from default settings.

Here's the Avisynth script:

LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\LoadDll.dll")
LoadCPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\ffms2.dll")
LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\amDCT.dll")
LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\masktools2.dll")
LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\VariableBlur.dll")
LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\warpsharp.dll")
LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\RgTools.dll")
LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\svpflow1.dll")
LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\svpflow2.dll")
LoadDLL("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\libfftw3f-3.dll")
Import("C:\Program Files\Hybrid\32bit\avisynthPlugins\LimitedSharpenFasterMod.avs")
Import("C:\Program Files\Hybrid\32bit\avisynthPlugins\InterFrame2.avsi")
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
# loading source: E:\USER\temp_dvd\Video\Test_Input\Test (Pre).mp4
#  input color sampling YV12
#  input luminance scale tv
FFVideoSource("E:\USER\temp_dvd\Video\TEST_I~1\TEST(P~1.MP4")
# current resolution: 1620x1080
# deblocking
# deblocking using amDCT
# padding to mod16 for speed
# Current resilution: 1632x1088
AddBorders(6,4,6,4)
amDCT(adapt=31,shift=5,quality=4,quant=9,matrix=8,expand=16,sharpWpos=6,sharpTpos=7,sharpWAmt=24,sharpTAmt=31,brightStart=205,brightAmt=30,ncpu=4)
# removing mod16 padding
Crop(6,4,-6,-4)
# sharpening
LSFmod(Smode=5,Smethod=3,Lmode=4,soft=-2,edgemaskHQ=true)
InterFrame(NewNum=60,NewDen=1,OverrideAlgo=23,Cores=6)
# letterboxing
AddBorders(150,0,150,0)
# Current resolution: 1920x1080
PreFetch(12)
return last

x264:

x264 --preset veryslow --crf 16.00 --profile high --level 4.2 --ref 4 --b-adapt 1 --vbv-maxrate 62500 --vbv-bufsize 78125 --threads 6 --sar 1:1 --qpfile GENERATED_QP_FILE --non-deterministic --range tv --colormatrix bt709 --demuxer raw --input-res 1920x1080 --input-csp i420 --input-range tv --input-depth 8 --fps 60/1 --output-depth 8 --output "C:\Users\Administrator\AppData\Local\Temp\Test (Post).264" -

Before every job, I did Config -> Reset All, and Hybrid restarts. I then set Config -> Path -> Default output path, and Internals -> Create debug file, level: 9.


01_HybridDebugOutput_CRASH_Default_Mem.txt:

FFmpeg crash

02_HybridDebugOutput_STALL_3072_NoDoub_Mem.txt:

CPU stall, all activity stopped

03_HybridDebugOutput_CRASH_1536_Mem_Nodistrib.txt:

FFmpeg crash

04_HybridDebugOutput_SLOW_Misc_Def.txt:

User stopped due to job running extremely slow

05_HybridDebugOutput_CRASH_LibavVideoSource_1536_Mem.txt:

FFmpeg crash

Unable to attach Logs.zip, using external source: https://bit.ly/2PlaWaL

[Selur]

Threading costs lots of RAM.
So if the whole thing get's instable when you use more threads it's probably a memory limit issue.
Avisynth+ how Hybrid uses it, is 32bit only and thus is limited to <4GB RAM usage.
Vapoursynth on the other hand is 64bit and thus not so limited in regards of RAM.

You can use AVSMeter (https://forum.doom9.org/showthread.php?t=174797), simply copy the AVSMeter.exe into your Hybrid/32bit folder and it should work fine and call it inside a Windows Command Prompt.

Disabling 'add distributor()' is an Avisynth MT option, by default Avisynth+ is used so that won't do a thing.

Assuming you got AVSMeter running, try whether it helps if you add:

SetMemoryMax(768)

Cu Selur

Ps.: As side notes:
a. in theory one could use Avisynth filters inside Vapoursynth, problem is there isn't a 64bit version of amDCT so that wouldn't help either.
b. higher cpu usage with Avisynth doesn't necessary have to mean higher speed
c. wondering why FFVideoSource is used for your source, when 'LibavVideoSource instead of FFmpegSource' is enabled.
d. Instead of using FFmpeg as Avisynth Script interpreter, you could also try avs2yuv (by disabling 'Prefer FFmpeg as decoder' and enabling 'Prefer avs2yub to decoder') this might lower the memory usage a tiny bit. (will probably not really fix the issue)

[The_Tinkerer]

AVSMeter.exe is already in the Hybrid\32bit folder. It looks like it's included as part of the Hybrid install.

Unfortunately I'm not savvy enough to figure out how to use it. Can you give me more info as to how to use it to add "SetMemoryMax(768)" as you describe? Do I do this somewhere within Hybrid, or do I run AVSMeter.exe separately?

I did try searching and all I could find is a reference to AVSMeter being used for the "Limit benchmark (%)" option. But even though that is checked by default, I don't see AVSMeter.exe in task manager as a running process when a job is running.

I also tried avs2yuv so that it runs instead of ffmpeg, but that just crashed the same way as ffmpeg does, so you're right, that didn't fix it.

[Selur]

a. yes, AvsMeter comes with Hybrid, but it isn't used during the job processing. It's only used to benchmak avisynth scripts. (Filtering->Benchmark) The AvsMeter version which comes with Hybrid probably isn't up-to-date.
b. You don't use AvsMeter to add that part you edit your script and call it with AvsMeter inside a command prompt.

"Path to AvsMeter"" "PATH TO YOUR AVISYNTH FILE"

AvsMeter shows the Memory usage, number of threads used etc. which helps to track down problems.

Cu Selur

[The_Tinkerer]

Understood.

The AVSMeter version included with Hybrid is up to date.

It also crashed.


With the Avisynth script direct from Hybrid:

C:\Program Files\Hybrid\32bit>avsmeter "TestScript.avs"

AVSMeter 2.9.0 (x86) - Copyright (c) 2012-2019, Groucho2004
AviSynth+ 0.1 (r2772, MT, i386) (0.1.0.0)

Number of frames:                25724
Length (hh:mm:ss.ms):     00:07:08.733
Frame width:                      1920
Frame height:                     1080
Framerate:                      60.000 (60/1)
Colorspace:                       i420

Frame (current | last):         8122 | 25723
FPS (cur | min | max | avg):    4.186 | 0.162 | 243901 | 4.452
Process memory usage:           1327 MiB
Thread count:                   40
CPU usage (current | average):  93.6% | 89.1%

Time (elapsed | estimated):     00:30:24.176 | 01:36:17.530


Exception 0xC0000005 [STATUS_ACCESS_VIOLATION]
Module:   C:\Windows\syswow64\KERNELBASE.dll
Address:  0x7660C5AF

C:\Program Files\Hybrid\32bit>

With the same script with SetMemoryMax(768) added to the beginning:

C:\Program Files\Hybrid\32bit>avsmeter "TestScript.avs"

AVSMeter 2.9.0 (x86) - Copyright (c) 2012-2019, Groucho2004
AviSynth+ 0.1 (r2772, MT, i386) (0.1.0.0)

Number of frames:                25724
Length (hh:mm:ss.ms):     00:07:08.733
Frame width:                      1920
Frame height:                     1080
Framerate:                      60.000 (60/1)
Colorspace:                       i420

Frame (current | last):         5839 | 25723
FPS (cur | min | max | avg):    1.506 | 0.258 | 238100 | 4.247
Process memory usage:           1145 MiB
Thread count:                   33
CPU usage (current | average):  88.1% | 89.1%

Time (elapsed | estimated):     00:22:54.771 | 01:40:56.620


Exception 0xC0000005 [STATUS_ACCESS_VIOLATION]
Module:   C:\Windows\syswow64\KERNELBASE.dll
Address:  0x7660C5AF

C:\Program Files\Hybrid\32bit>

Any other ideas? 

[The_Tinkerer]

I tried doing a SetMemoryMax(512) and it still crashed. I'm not sure that this is an issue with too much memory being used in a x86 environment. I suspect it's amDCT bugginess. I don't think the author's updated it for quite some time. It's too bad there's not a good equivalent to it in Vapoursynth.

[Selur]

According to the numbers AVSMeter provides it isn't an issue with the memory allocation.
< 1.4GB is definitely save.

This looks like:
a. a bug in a filter (try whether this also happens if you disable amDCT and/or LSFmod and/or Interframe to narrow down which filter causes the problem)
b. something interfering with avisynth filter (some overzealous firewall or av solution? <- try disabling those while testing)
c. some problem with the memory

Cu Selur

[The_Tinkerer]

Yeah I forgot to mention the only Anti Virus I'm using is built in Windows Defender. And also built in Windows Firewall.

Thanks for taking the time to respond so quickly, I appreciate it.

[Selur]

So it's either one of the used filters or a memory problem.

Cu Selur

[The_Tinkerer]

Thanks for the help. I've narrowed the problem down to amDCT. Not sure if using it with a different version of Avisynth would make a difference. There's a support thread but I don't have a Doom9 account and they make new users wait 5 days before being able to post, to keep out spammers. (Also keeping out legit users, but oh well.)

The author did release a "Testing" version 1.1 update in January 2019, I tried it and it still crashed.