Selur's Little Message Board

Pages: 1 2

I don't expect anyone to be able to figure this out due to the highly idiosyncratic and esoteric nature of the problem. But, I have wasted days on this now and I am tearing my hair out trying everything to troubleshoot this, to no avail.

The problem is, when I try to run jobs using Hybrid with Avisynth, I get flaky and unstable behavior. Sometimes, FFmpeg randomly crashes at random points in the job. Sometimes, all CPU activity stalls with no crash, but the job is now stuck. I can usually reset and restart the job and it will complete the second or third time.

[Image: Co9GUIE.png]

Code:
Log Name:      Application

Source:        Application Error

Date:          4/18/2019 10:54:56 PM

Event ID:      1000

Task Category: (100)

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      WIN-XXXXXXXXXXX

Description:

Faulting application name: ffmpeg.exe, version: 0.0.0.0, time stamp: 0x00000000

Faulting module name: KERNELBASE.dll, version: 6.1.7601.24408, time stamp: 0x5c92f101

Exception code: 0xc0000005

Fault offset: 0x0000c5af

Faulting process id: 0x13b4

Faulting application start time: 0x01d4f658174d46a1

Faulting application path: C:\PROGRA~1\Hybrid\32bit\ffmpeg.exe

Faulting module path: C:\Windows\syswow64\KERNELBASE.dll

Report Id: 82d22dc1-624e-11e9-b816-0050560b8935

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="Application Error" />

    <EventID Qualifiers="0">1000</EventID>

    <Level>2</Level>

    <Task>100</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2019-04-19T02:54:56.000000000Z" />

    <EventRecordID>249111</EventRecordID>

    <Channel>Application</Channel>

    <Computer>WIN-XXXXXXXXXXX</Computer>

    <Security />

  </System>

  <EventData>

    <Data>ffmpeg.exe</Data>

    <Data>0.0.0.0</Data>

    <Data>00000000</Data>

    <Data>KERNELBASE.dll</Data>

    <Data>6.1.7601.24408</Data>

    <Data>5c92f101</Data>

    <Data>c0000005</Data>

    <Data>0000c5af</Data>

    <Data>13b4</Data>

    <Data>01d4f658174d46a1</Data>

    <Data>C:\PROGRA~1\Hybrid\32bit\ffmpeg.exe</Data>

    <Data>C:\Windows\syswow64\KERNELBASE.dll</Data>

    <Data>82d22dc1-624e-11e9-b816-0050560b8935</Data>

  </EventData>

</Event>

I now have a job that I am unable to complete without a crash or stall.

I have to set the number of MT threads to 12 in the Avisynth Misc tab or the job will run WAY too slow. All 6 cores are used, but the CPU utilization is never more than 20% and the performance is "spiky" in Task Manager, meaning there are only short "bursts" or spikes of activity in between no activity on all cores.

In addition to that, sometimes when starting Hybrid I get random crashes; all I can do is close and restart. It usually runs on restart. Sometimes I will get two crashes in a row until restarting successfully.

[Image: 9xBQXjF.png]

Code:
Log Name:      Application

Source:        Application Error

Date:          4/18/2019 11:36:10 PM

Event ID:      1000

Task Category: (100)

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      WIN-XXXXXXXXXXX

Description:

Faulting application name: Hybrid.exe, version: 2019.3.13.0, time stamp: 0x5c89658b

Faulting module name: Qt5Gui.dll, version: 5.12.0.0, time stamp: 0x5bee693c

Exception code: 0xc00000fd

Fault offset: 0x0000000000109b7c

Faulting process id: 0x764

Faulting application start time: 0x01d4f66106a85341

Faulting application path: C:\Program Files\Hybrid\Hybrid.exe

Faulting module path: C:\Program Files\Hybrid\Qt5Gui.dll

Report Id: 457021c1-6254-11e9-b816-0050560b8935

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="Application Error" />

    <EventID Qualifiers="0">1000</EventID>

    <Level>2</Level>

    <Task>100</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2019-04-19T03:36:10.000000000Z" />

    <EventRecordID>249116</EventRecordID>

    <Channel>Application</Channel>

    <Computer>WIN-XXXXXXXXXXX</Computer>

    <Security />

  </System>

  <EventData>

    <Data>Hybrid.exe</Data>

    <Data>2019.3.13.0</Data>

    <Data>5c89658b</Data>

    <Data>Qt5Gui.dll</Data>

    <Data>5.12.0.0</Data>

    <Data>5bee693c</Data>

    <Data>c00000fd</Data>

    <Data>0000000000109b7c</Data>

    <Data>764</Data>

    <Data>01d4f66106a85341</Data>

    <Data>C:\Program Files\Hybrid\Hybrid.exe</Data>

    <Data>C:\Program Files\Hybrid\Qt5Gui.dll</Data>

    <Data>457021c1-6254-11e9-b816-0050560b8935</Data>

  </EventData>

</Event>

I seem to be able to run Vapoursynth without issue.

The reason I am using Avisynth is I am using amDCT Deblocker, which is not available in Vapoursynth. I suspect that this filter is what is causing most of the instability, but when it works it gives me the best results for what it does.

Do you have ANY idea on how to go about troubleshooting this?

I did try replacing C:\Program Files\Hybrid\32bit\ffmpeg.exe with ffmpeg.exe x86 version 4.1.1 (the latest release build), but it had no effect.

CPU-Z:
[Image: b6cTcaA.png]

CPUZ.txt in Logs.zip

I have 5 logs. The only thing different between the logs are settings on the Avisynth Misc tab. Red boxes show what is changed from default settings.

Here's the Avisynth script:

Code:
LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\LoadDll.dll")

LoadCPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\ffms2.dll")

LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\amDCT.dll")

LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\masktools2.dll")

LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\VariableBlur.dll")

LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\warpsharp.dll")

LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\RgTools.dll")

LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\svpflow1.dll")

LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\svpflow2.dll")

LoadDLL("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\libfftw3f-3.dll")

Import("C:\Program Files\Hybrid\32bit\avisynthPlugins\LimitedSharpenFasterMod.avs")

Import("C:\Program Files\Hybrid\32bit\avisynthPlugins\InterFrame2.avsi")

SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)

# loading source: E:\USER\temp_dvd\Video\Test_Input\Test (Pre).mp4

#  input color sampling YV12

#  input luminance scale tv

FFVideoSource("E:\USER\temp_dvd\Video\TEST_I~1\TEST(P~1.MP4")

# current resolution: 1620x1080

# deblocking

# deblocking using amDCT

# padding to mod16 for speed

# Current resilution: 1632x1088

AddBorders(6,4,6,4)

amDCT(adapt=31,shift=5,quality=4,quant=9,matrix=8,expand=16,sharpWpos=6,sharpTpos=7,sharpWAmt=24,sharpTAmt=31,brightStart=205,brightAmt=30,ncpu=4)

# removing mod16 padding

Crop(6,4,-6,-4)

# sharpening

LSFmod(Smode=5,Smethod=3,Lmode=4,soft=-2,edgemaskHQ=true)

InterFrame(NewNum=60,NewDen=1,OverrideAlgo=23,Cores=6)

# letterboxing

AddBorders(150,0,150,0)

# Current resolution: 1920x1080

PreFetch(12)

return last

x264:

Code:
x264 --preset veryslow --crf 16.00 --profile high --level 4.2 --ref 4 --b-adapt 1 --vbv-maxrate 62500 --vbv-bufsize 78125 --threads 6 --sar 1:1 --qpfile GENERATED_QP_FILE --non-deterministic --range tv --colormatrix bt709 --demuxer raw --input-res 1920x1080 --input-csp i420 --input-range tv --input-depth 8 --fps 60/1 --output-depth 8 --output "C:\Users\Administrator\AppData\Local\Temp\Test (Post).264" -

Before every job, I did Config -> Reset All, and Hybrid restarts. I then set Config -> Path -> Default output path, and Internals -> Create debug file, level: 9.

01_HybridDebugOutput_CRASH_Default_Mem.txt:
[Image: n2tHF63.png]

FFmpeg crash

02_HybridDebugOutput_STALL_3072_NoDoub_Mem.txt:
[Image: Ot81Qbj.png]

CPU stall, all activity stopped

03_HybridDebugOutput_CRASH_1536_Mem_Nodistrib.txt:
[Image: vWq6iC9.png]

FFmpeg crash

04_HybridDebugOutput_SLOW_Misc_Def.txt:
[Image: XRwagkn.png]

User stopped due to job running extremely slow

05_HybridDebugOutput_CRASH_LibavVideoSource_1536_Mem.txt:
[Image: xFkkAhV.png]

FFmpeg crash

Unable to attach Logs.zip, using external source: https://bit.ly/2PlaWaL

Threading costs lots of RAM. Smile

So if the whole thing get's instable when you use more threads it's probably a memory limit issue.
Avisynth+ how Hybrid uses it, is 32bit only and thus is limited to <4GB RAM usage.
Vapoursynth on the other hand is 64bit and thus not so limited in regards of RAM.

You can use AVSMeter (https://forum.doom9.org/showthread.php?t=174797), simply copy the AVSMeter.exe into your Hybrid/32bit folder and it should work fine and call it inside a Windows Command Prompt.

Disabling 'add distributor()' is an Avisynth MT option, by default Avisynth+ is used so that won't do a thing. Smile

Assuming you got AVSMeter running, try whether it helps if you add:

Code:
SetMemoryMax(768)

Cu Selur

Ps.: As side notes:
a. in theory one could use Avisynth filters inside Vapoursynth, problem is there isn't a 64bit version of amDCT so that wouldn't help either.
b. higher cpu usage with Avisynth doesn't necessary have to mean higher speed
c. wondering why FFVideoSource is used for your source, when 'LibavVideoSource instead of FFmpegSource' is enabled.
d. Instead of using FFmpeg as Avisynth Script interpreter, you could also try avs2yuv (by disabling 'Prefer FFmpeg as decoder' and enabling 'Prefer avs2yub to decoder') this might lower the memory usage a tiny bit. (will probably not really fix the issue)

AVSMeter.exe is already in the Hybrid\32bit folder. It looks like it's included as part of the Hybrid install.

Unfortunately I'm not savvy enough to figure out how to use it. Can you give me more info as to how to use it to add "SetMemoryMax(768)" as you describe? Do I do this somewhere within Hybrid, or do I run AVSMeter.exe separately?

I did try searching and all I could find is a reference to AVSMeter being used for the "Limit benchmark (%)" option. But even though that is checked by default, I don't see AVSMeter.exe in task manager as a running process when a job is running.

I also tried avs2yuv so that it runs instead of ffmpeg, but that just crashed the same way as ffmpeg does, so you're right, that didn't fix it.

a. yes, AvsMeter comes with Hybrid, but it isn't used during the job processing. It's only used to benchmak avisynth scripts. (Filtering->Benchmark) The AvsMeter version which comes with Hybrid probably isn't up-to-date.
b. You don't use AvsMeter to add that part you edit your script and call it with AvsMeter inside a command prompt.

Code:
"Path to AvsMeter"" "PATH TO YOUR AVISYNTH FILE"

AvsMeter shows the Memory usage, number of threads used etc. which helps to track down problems.

Cu Selur

Understood.

The AVSMeter version included with Hybrid is up to date. [Image: thumbsup.png]

It also crashed.

With the Avisynth script direct from Hybrid:

Code:
C:\Program Files\Hybrid\32bit>avsmeter "TestScript.avs"

AVSMeter 2.9.0 (x86) - Copyright (c) 2012-2019, Groucho2004

AviSynth+ 0.1 (r2772, MT, i386) (0.1.0.0)

Number of frames:                25724

Length (hh:mm:ss.ms):     00:07:08.733

Frame width:                      1920

Frame height:                     1080

Framerate:                      60.000 (60/1)

Colorspace:                       i420

Frame (current | last):         8122 | 25723

FPS (cur | min | max | avg):    4.186 | 0.162 | 243901 | 4.452

Process memory usage:           1327 MiB

Thread count:                   40

CPU usage (current | average):  93.6% | 89.1%

Time (elapsed | estimated):     00:30:24.176 | 01:36:17.530

Exception 0xC0000005 [STATUS_ACCESS_VIOLATION]

Module:   C:\Windows\syswow64\KERNELBASE.dll

Address:  0x7660C5AF

C:\Program Files\Hybrid\32bit>

With the same script with SetMemoryMax(768) added to the beginning:

Code:
C:\Program Files\Hybrid\32bit>avsmeter "TestScript.avs"

AVSMeter 2.9.0 (x86) - Copyright (c) 2012-2019, Groucho2004

AviSynth+ 0.1 (r2772, MT, i386) (0.1.0.0)

Number of frames:                25724

Length (hh:mm:ss.ms):     00:07:08.733

Frame width:                      1920

Frame height:                     1080

Framerate:                      60.000 (60/1)

Colorspace:                       i420

Frame (current | last):         5839 | 25723

FPS (cur | min | max | avg):    1.506 | 0.258 | 238100 | 4.247

Process memory usage:           1145 MiB

Thread count:                   33

CPU usage (current | average):  88.1% | 89.1%

Time (elapsed | estimated):     00:22:54.771 | 01:40:56.620

Exception 0xC0000005 [STATUS_ACCESS_VIOLATION]

Module:   C:\Windows\syswow64\KERNELBASE.dll

Address:  0x7660C5AF

C:\Program Files\Hybrid\32bit>

Any other ideas? Undecided

I tried doing a SetMemoryMax(512) and it still crashed. I'm not sure that this is an issue with too much memory being used in a x86 environment. I suspect it's amDCT bugginess. I don't think the author's updated it for quite some time. It's too bad there's not a good equivalent to it in Vapoursynth.

According to the numbers AVSMeter provides it isn't an issue with the memory allocation.
< 1.4GB is definitely save.

This looks like:
a. a bug in a filter (try whether this also happens if you disable amDCT and/or LSFmod and/or Interframe to narrow down which filter causes the problem)
b. something interfering with avisynth filter (some overzealous firewall or av solution? <- try disabling those while testing)
c. some problem with the memory

Cu Selur

Yeah I forgot to mention the only Anti Virus I'm using is built in Windows Defender. And also built in Windows Firewall.

Thanks for taking the time to respond so quickly, I appreciate it. Smile

So it's either one of the used filters or a memory problem.

Cu Selur

Thanks for the help. I've narrowed the problem down to amDCT. Not sure if using it with a different version of Avisynth would make a difference. There's a support thread but I don't have a Doom9 account and they make new users wait 5 days before being able to post, to keep out spammers. (Also keeping out legit users, but oh well.) Tongue

The author did release a "Testing" version 1.1 update in January 2019, I tried it and it still crashed. Tongue

Pages: 1 2

The_Tinkerer

Selur

The_Tinkerer

Selur

The_Tinkerer

The_Tinkerer

Selur

The_Tinkerer

Selur

The_Tinkerer