Out of bounds memory error in VirtualDub2 - lostsnowboardvideos - 12.07.2024
Any VDub2 guru's in here? Recently installed avisynth+ x64 and have had about 50% success converting mkv files to mp4 through virtualdub2 (using QTGMC deinterlacer for interlaced content and the reason I switched to using Hyrbid). The other portion of the time is spent ripping my hair out researching the out of bounds memory error fixes. I have attached the crash log and a simple avs script I am running.
I'm running on 6 cores, 12 logical processors, windows 10 64-bit desktop, 32GB. The out of bounds memory access violation keeps occurring in the x264 codec. I've tried lowering the prefetch to 8 and 6 to see if that helps but it didn't. Also tried reducing the EdiThreads too. One interesting note is I am running the exact same scripts and programs on a laptop as I am on my desktop. The laptop is windows 11 but significantly slower, 4 cores, 8 LP and have had zero issues running scripts. Problem is one video takes about 6-7x longer to encode.
Input file is an interlaced mpeg2 mkv 29.97 ripped with MakeMKV. Output to progressive 59.94
Error code file:
VirtualDub2 crash report -- build 1 (release-AMD64)
--------------------------------------
Disassembly:
64c4ba80: f6e3 mul al, bl
64c4ba82: c5 db 0c5h
64c4ba83: d5f6 aad 0f6h
64c4ba85: ebc5 jmp 7ff764c4ba4c
64c4ba87: cdf6 int 0f6h
64c4ba89: f3c5 db 0c5h
64c4ba8b: fd std
64c4ba8c: fd std
64c4ba8d: c4 db 0c4h
64c4ba8e: c5 db 0c5h
64c4ba8f: f5 cmc
64c4ba90: fd std
64c4ba91: cdc5 int 0c5h
64c4ba93: ed in eax, dx
64c4ba94: fd std
64c4ba95: d6 db 0d6h
64c4ba96: 4c8b5c2430 mov r11, [rsp+30h]
64c4ba9b: c5 db 0c5h
64c4ba9c: fd std
64c4ba9d: 6bc1c5 imul eax, ecx, 0c5h
64c4baa0: ed in eax, dx
64c4baa1: 6bd2c4 imul edx, edx, 0c4h
64c4baa4: e27d loop 7ff764c4bb23
64c4baa6: 02c2 add al, dl
64c4baa8: c4 db 0c4h
64c4baa9: e37d jecxz7dh
64c4baab: 39c1 cmp ecx, eax
64c4baad: 01c5 add ebp, eax
64c4baaf: f9 stc
64c4bab0: fec1 inc cl
64c4bab2: c4 db 0c4h
64c4bab3: c1797f03 sar dword ptr [rcx+7fh], 03h
64c4bab7: c5 db 0c5h
64c4bab8: f8 clc
64c4bab9: 28742408 sub [rsp+08h], dh
64c4babd: c5 db 0c5h
64c4babe: f8 clc
64c4babf: 77c3 ja 7ff764c4ba84
64c4bac1: 0f db 0fh
64c4bac2: 1f db 1fh
64c4bac3: 8400 test [rax], al
64c4bac5: 0000 add [rax], al
64c4bac7: 0000 add [rax], al
64c4bac9: 0f db 0fh
64c4baca: 1f db 1fh
64c4bacb: 800000 add byte ptr [rax], 00h
64c4bace: 0000 add [rax], al
64c4bad0: c5 db 0c5h
64c4bad1: f8 clc
64c4bad2: 29742408 sub [rsp+08h], esi
64c4bad6: c5 db 0c5h
64c4bad7: f8 clc
64c4bad8: 297c2418 sub [rsp+18h], edi
64c4badc: 4c8b542428 mov r10, [rsp+28h]
64c4bae1: 4c8b5c2430 mov r11, [rsp+30h]
64c4bae6: 4b8d045b lea rax, [r11+r11*2]
64c4baea: c4 db 0c4h
64c4baeb: e27d loop 7ff764c4bb6a
64c4baed: 5a pop edx
64c4baee: 21c4 and esp, eax
64c4baf0: e27d loop 7ff764c4bb6f
64c4baf2: 5a pop edx
64c4baf3: 6910c5fa6f02 imul edx, [rax], 026ffac5
64c4baf9: c4 db 0c4h
64c4bafa: c17a6f08 sar dword ptr [rdx+6fh], 08h
64c4bafe: c4 db 0c4h
64c4baff: a17a6f141a mov eax, [1a146f7a]
64c4bb04: c4 db 0c4h
64c4bb05: 817a6f1c18c4c3 cmp dword ptr [rdx+6fh], c3c4181c
64c4bb0c: 7d38 jge 7ff764c4bb46
64c4bb0e: 0101 add [rcx], eax
64c4bb10: c4 db 0c4h <-- FAULT
64c4bb11: c3 ret
64c4bb12: 7538 jnz 7ff764c4bb4c
64c4bb14: 0a01 or al, [rcx]
64c4bb16: c4 db 0c4h
64c4bb17: 836d3814 sub dword ptr [rbp+38h], 14h
64c4bb1b: 1901 sbb [rcx], eax
64c4bb1d: c4 db 0c4h
64c4bb1e: 8365381c and dword ptr [rbp+38h], 1ch
64c4bb22: 1a01 sbb al, [rcx]
64c4bb24: c5 db 0c5h
64c4bb25: fd std
64c4bb26: f6c4c5 test ah, 0c5h
64c4bb29: f5 cmc
64c4bb2a: f6 db 0f6h
64c4bb2b: cc int 3
64c4bb2c: c5 db 0c5h
64c4bb2d: ed in eax, dx
64c4bb2e: f6d5 not ch
64c4bb30: c5 db 0c5h
64c4bb31: e5f6 in eax, 0f6h
64c4bb33: ddc5 ffree st(5)
64c4bb35: fd std
64c4bb36: fd std
64c4bb37: c2c5f5 ret f5c5
64c4bb3a: fd std
64c4bb3b: cb retf
64c4bb3c: c4 db 0c4h
64c4bb3d: e27d loop 7ff764c4bbbc
64c4bb3f: 5a pop edx
64c4bb40: 7120 jno 7ff764c4bb62
64c4bb42: c4 db 0c4h
64c4bb43: e27d loop 7ff764c4bbc2
64c4bb45: 5a pop edx
64c4bb46: 7930 jns 7ff764c4bb78
64c4bb48: c4 db 0c4h
64c4bb49: a17a6f145a mov eax, [5a146f7a]
64c4bb4e: c4 db 0c4h
64c4bb4f: 817a6f1c58c5fa cmp dword ptr [rdx+6fh], fac5581c
64c4bb56: 6f outsd
64c4bb57: 2402 and al, 02h
64c4bb59: c4 db 0c4h
64c4bb5a: c17a6f2c sar dword ptr [rdx+6fh], 2ch
64c4bb5e: 00c4 add ah, al
64c4bb60: 836d3814 sub dword ptr [rbp+38h], 14h
64c4bb64: 59 pop ecx
64c4bb65: 01c4 add esp, eax
64c4bb67: 8365381c and dword ptr [rbp+38h], 1ch
64c4bb6b: 5a pop edx
64c4bb6c: 01c4 add esp, eax
64c4bb6e: c3 ret
64c4bb6f: 5d pop ebp
64c4bb70: 382401 cmp [rcx+rax], ah
64c4bb73: 01c4 add esp, eax
64c4bb75: c3 ret
64c4bb76: 55 push ebp
64c4bb77: 382c02 cmp [rdx+rax], ch
64c4bb7a: 01c5 add ebp, eax
64c4bb7c: ed in eax, dx
64c4bb7d: f6d6 not dh
64c4bb7f: c5 db 0c5h
Built on R-COMP on Sun May 26 18:00:29 2024 using compiler version 1939
Windows 10 x64
Memory status: virtual free 134163245M/134217728M, commit limit 37582M, physical total 32718M
RAX = 11a0
RBX = 19217ecbbcf
RCX = 19126f8ef40
RDX = 19217ecb00b
RSI = ffffffffffffffff
RDI = ffffffff
RBP = 19126f88b00
R8 = 19217ecb013
R9 = 19217ecb5eb
R10 = 19317ecb00e
R11 = 5e0
R12 = fffffffffffffffc
R13 = 4
R14 = bc0
R15 = 19217ecc1b1
RSP = 4dc50f6e68
RIP = 64c4bb10
EFLAGS = 00010216
Crash reason: Access Violation
Crash context:
An out-of-bounds memory access (access violation) occurred in module 'x264-8'...
...reading address 0000019317ECB00E.
Pointer dumps:
RBX 19217ecbbcb: b7b7b6b5 b7b7b6b7 b7b7b7b7 b9b8b7b7 bcbcbbba b6b8bdbd bab9b8b6 bbbbbbbb
RCX 19126f8ef40: b6b7b8ba b3b4b5b6 b5b4b3b3 babab8b7 b8b8b9ba b6b7b8b8 b7b6b5b6 bcbcbbb9
RDX 19217ecb00b: b7b8b7b6 b3b4b5b6 b4b2b1b1 b8b6b5b5 b8b7b6b7 b5b7bab9 b9b9b8b6 bbbbbaba
RSP 4dc50f6e68: 64be5700 00000000 00000000 00000000 00000000 00000000 00000000 00000000
4dc50f6e88: 00000000 00000000 17ecb00e 00000193 000005e0 00000000 c50f70e0 0000004d
4dc50f6ea8: 00000010 00000000 64c84500 00000000 ffffffff ffffffff 26f8ef40 00000191
4dc50f6ec8: 000005e0 00000000 ffffffff 0000004d 17ecaa30 00000192 26f22280 00000191
RBP 19126f88b00: 0010f3ff 00000012 00000003 00000000 00000001 00000000 00000009 000005a0
19126f88b20: 00000438 00000002 00000008 00000032 000244d2 00000000 00000001 00000001
19126f88b40: 00000000 00000005 00000000 00000002 00000002 00000001 00000000 00000010
19126f88b60: 00000001 000000fa 00000019 00000028 00000000 00000008 00000002 00000000
R8 19217ecb013: b4b2b1b1 b8b6b5b5 b8b7b6b7 b5b7bab9 b9b9b8b6 bbbbbaba bababbbb babababb
R9 19217ecb5eb: b7b8b7b5 b5b6b6b7 b5b4b3b4 b7b7b7b6 babab8b7 b5b7bbba bab9b7b5 bbbbbabb
R15 19217ecc1b1: b9b9b8b7 bababab9 bdbdbcbb bdbebdbd b8b7b8ba bbbabab9 bdbcbbbb bcbcbcbd
Thread call stack:
64c4bb10: x264-8!VDDriverProc [64b40000+9290+102880]
64bcf6a2: x264-8!VDDriverProc [64b40000+9290+86412]
64c3f593: x264-8!VDDriverProc [64b40000+9290+f6303]
64b67b49: x264-8!VDDriverProc [64b40000+9290+1e8b9]
64bbe5b6: x264-8!VDDriverProc [64b40000+9290+75326]
64c3ffb5: x264-8!VDDriverProc [64b40000+9290+f6d25]
64bd7781: x264-8!VDDriverProc [64b40000+9290+8e4f1]
64c1e86a: x264-8!VDDriverProc [64b40000+9290+d55da]
64c1d9b4: x264-8!VDDriverProc [64b40000+9290+d4724]
64b8acae: x264-8!VDDriverProc [64b40000+9290+41a1e]
64c1a440: x264-8!VDDriverProc [64b40000+9290+d11b0]
64c01cc7: x264-8!VDDriverProc [64b40000+9290+b8a37]
64c236f1: x264-8!VDDriverProc [64b40000+9290+da461]
64c09b1c: x264-8!VDDriverProc [64b40000+9290+c088c]
64bd37e6: x264-8!VDDriverProc [64b40000+9290+8a556]
64b58a6f: x264-8!VDDriverProc [64b40000+9290+f7df]
7ffd16b4cb51: ntdll!RtlFindActivationContextSectionString [7ffd16b30000+1c6c0+491]
7ffd16b4d23b: ntdll!RtlFindActivationContextSectionGuid [7ffd16b30000+1cd80+4bb]
7ffd16b4c7bc: ntdll!RtlFindActivationContextSectionString [7ffd16b30000+1c6c0+fc]
7ffd16b4c3c8: ntdll!RtlInitUnicodeStringEx [7ffd16b30000+1c270+158]
7ffcdc2e4517: FileSyncShell64!?SizeUnknown@QoS@@YAIXZ [7ffcdc2e0000+43e0+137]
7ffd16b4aced: ntdll!RtlDosApplyFileIsolationRedirection_Ustr [7ffd16b30000+1a9e0+30d]
7ffd16b51374: ntdll!RtlEqualUnicodeString [7ffd16b30000+20d10+664]
7ffd16b91953: ntdll!RtlDetectHeapLeaks [7ffd16b30000+61770+1e3]
7ffd16b49877: ntdll!RtlDeactivateActivationContextUnsafeFast [7ffd16b30000+197b0+c7]
7ffd16b4f5f7: ntdll!RtlGetFullPathName_UstrEx [7ffd16b30000+1d890+1d67]
7ffd16b4fcf4: ntdll!RtlGetFullPathName_UstrEx [7ffd16b30000+1d890+2464]
7ffd16b46dad: ntdll!RtlDosPathNameToNtPathName_U_WithStatus [7ffd16b30000+16be0+1cd]
7ffd16b46cdf: ntdll!RtlDosPathNameToNtPathName_U_WithStatus [7ffd16b30000+16be0+ff]
7ffd16b46d4b: ntdll!RtlDosPathNameToNtPathName_U_WithStatus [7ffd16b30000+16be0+16b]
7ffd16b5121f: ntdll!RtlEqualUnicodeString [7ffd16b30000+20d10+50f]
7ffd16b5b44d: ntdll!RtlAllocateHeap [7ffd16b30000+2a9a0+aad]
7ffd16b4a8fc: ntdll!RtlReleaseRelativeName [7ffd16b30000+1a320+5dc]
7ffd16b4a407: ntdll!RtlReleaseRelativeName [7ffd16b30000+1a320+e7]
7ffd16b5df6f: ntdll!RtlAllocateHeap [7ffd16b30000+2a9a0+35cf]
7ffd16b4f47e: ntdll!RtlGetFullPathName_UstrEx [7ffd16b30000+1d890+1bee]
7ffd16b50b31: ntdll!RtlTryEnterCriticalSection [7ffd16b30000+207e0+351]
7ffd16b5b90b: ntdll!RtlAllocateHeap [7ffd16b30000+2a9a0+f6b]
7ffd16b5b3c7: ntdll!RtlAllocateHeap [7ffd16b30000+2a9a0+a27]
7ffd16b602c9: ntdll!LdrGetProcedureAddressForCaller [7ffd16b30000+2fdc0+509]
7ffcdc3882b4: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+4cd54]
7ffcdc387ff6: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+4ca96]
7ffd0fe029ff: WTSAPI32!IsInteractiveUserSession [7ffd0fe00000+2630+3cf]
7ffcdc388531: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+4cfd1]
7ffcdc387d49: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+4c7e9]
7ffcdc36a7c6: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+2f266]
7ffcdc36b762: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+30202]
7ffd16b49877: ntdll!RtlDeactivateActivationContextUnsafeFast [7ffd16b30000+197b0+c7]
7ffcdc36b16d: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+2fc0d]
7ffd16b49a26: ntdll!RtlActivateActivationContextUnsafeFast [7ffd16b30000+19900+126]
7ffd16b49993: ntdll!RtlActivateActivationContextUnsafeFast [7ffd16b30000+19900+93]
7ffcdc36b3bb: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+2fe5b]
7ffd16b49aff: ntdll!RtlActivateActivationContextUnsafeFast [7ffd16b30000+19900+1ff]
7ffd16b6e693: ntdll!RtlGetVersion [7ffd16b30000+3e4e0+1b3]
7ffd16b8eec4: ntdll!LdrEnumerateLoadedModules [7ffd16b30000+5edd0+f4]
7ffd16b47955: ntdll!LdrShutdownThread [7ffd16b30000+17450+505]
7ffd16b47877: ntdll!LdrShutdownThread [7ffd16b30000+17450+427]
7ffd16b478aa: ntdll!LdrShutdownThread [7ffd16b30000+17450+45a]
7ffd16b4791d: ntdll!LdrShutdownThread [7ffd16b30000+17450+4cd]
7ffd16b4793d: ntdll!LdrShutdownThread [7ffd16b30000+17450+4ed]
7ffd16ba60bd: ntdll!LdrInitializeThunk [7ffd16b30000+75c20+49d]
7ffd16ba5c83: ntdll!LdrInitializeThunk [7ffd16b30000+75c20+63]
7ffd16ba5c38: ntdll!LdrInitializeThunk [7ffd16b30000+75c20+18]
7ffd16b94821: ntdll!RtlWakeAddressAll [7ffd16b30000+646e0+141]
7ffd16b5fae2: ntdll!RtlEnterCriticalSection [7ffd16b30000+2faa0+42]
7ffd16b81910: ntdll!RtlRestoreLastWin32Error [7ffd16b30000+518d0+40]
7ffd16b95897: ntdll!RtlSleepConditionVariableCS [7ffd16b30000+65770+127]
7ffd143b478d: KERNELBASE!OpenThreadToken [7ffd14380000+34740+4d]
64b57416: x264-8!VDDriverProc [64b40000+9290+e186]
64b5b0d0: x264-8!VDDriverProc [64b40000+9290+11e40]
64b51b88: x264-8!VDDriverProc [64b40000+9290+88f8]
64b5195b: x264-8!VDDriverProc [64b40000+9290+86cb]
64b51a0a: x264-8!VDDriverProc [64b40000+9290+877a]
64b65f00: x264-8!VDDriverProc [64b40000+9290+1cc70]
7ffd1524aa7a: msvcrt!__threadid [7ffd15210000+3a800+27a]
64b51813: x264-8!VDDriverProc [64b40000+9290+8583]
7ffd1524af5a: msvcrt!_beginthreadex [7ffd15210000+3ae30+12a]
7ffd1524b02c: msvcrt!_endthreadex [7ffd15210000+3af80+ac]
7ffd15167344: KERNEL32!BaseThreadInitThunk [7ffd15150000+17330+14]
7ffd16b7cc91: ntdll!RtlUserThreadStart [7ffd16b30000+4cc70+21]
-- End of report
avs script:
SetFilterMTMode("qtgmc", 2)
FFmpegSource2("B1_t00.mkv", atrack=-1)
AssumeBFF()
QTGMC(Preset="slower", EdiThreads=3)
LanczosResize(1440,1080)
KNLMeansCL(d=1, a=2, s=4, h=1.2, wref=1.0, channels="auto", wmode=0, device_type="auto", mode_9_to_15bits=1, info=false)
Prefetch(10)
Edit to add: I had converted several files early on successfully and then this started happening and have 1 successful encode on it since I started getting the errors.
RE: Out of bounds memory error in VirtualDub2 - Selur - 12.07.2024
Sorry, I have no clue about VirtualDub and the x264 vfw component it uses.
Seeing that the out-of-memory occurs in the vfw x264, adjusting the memory usage in for Avisynth seems like the wrong approach.
Things I would recommend trying:- use AVSMeter to seem whether Avisynths memory consumption is the problem. (assuming your input video is SD, I doubt it)
If the memory issue stems from Avisynth, using a different source filter and other adjustments to the script might help.
- try using VirtualDubs external encoder with x264s cli encoder instead of the vfw interface to see whether it is a general problem in x264 or just x264s vfw interface. If it's the vfw interface, either don't use it or contact the developer. Additionally, deactivating different ASM optimizations, assuming the vfw interface allows it, might help.
Since you mentioned Hybrid, if you can provide proper details ([INFO] Infos needed to fix&reproduce bugs,..) and a source sample which allows reproducing the problem, I can look into that.
Cu Selur
|