Out of bounds memory error in VirtualDub2

[lostsnowboardvideos]

Any VDub2 guru's in here? Recently installed avisynth+ x64 and have had about 50% success converting mkv files to mp4 through virtualdub2 (using QTGMC deinterlacer for interlaced content and the reason I switched to using Hyrbid). The other portion of the time is spent ripping my hair out researching the out of bounds memory error fixes. I have attached the crash log and a simple avs script I am running.

I'm running on 6 cores, 12 logical processors, windows 10 64-bit desktop, 32GB. The out of bounds memory access violation keeps occurring in the x264 codec. I've tried lowering the prefetch to 8 and 6 to see if that helps but it didn't. Also tried reducing the EdiThreads too. One interesting note is I am running the exact same scripts and programs on a laptop as I am on my desktop. The laptop is windows 11 but significantly slower, 4 cores, 8 LP and have had zero issues running scripts. Problem is one video takes about 6-7x longer to encode.

Input file is an interlaced mpeg2 mkv 29.97 ripped with MakeMKV. Output to progressive 59.94

Error code file:

VirtualDub2 crash report -- build 1 (release-AMD64)
--------------------------------------

Disassembly:
64c4ba80: f6e3            mul    al, bl
64c4ba82: c5              db     0c5h
64c4ba83: d5f6            aad    0f6h
64c4ba85: ebc5            jmp    7ff764c4ba4c
64c4ba87: cdf6            int    0f6h
64c4ba89: f3c5            db     0c5h
64c4ba8b: fd              std   
64c4ba8c: fd              std   
64c4ba8d: c4              db     0c4h
64c4ba8e: c5              db     0c5h
64c4ba8f: f5              cmc   
64c4ba90: fd              std   
64c4ba91: cdc5            int    0c5h
64c4ba93: ed              in     eax, dx
64c4ba94: fd              std   
64c4ba95: d6              db     0d6h
64c4ba96: 4c8b5c2430      mov    r11, [rsp+30h]
64c4ba9b: c5              db     0c5h
64c4ba9c: fd              std   
64c4ba9d: 6bc1c5          imul   eax, ecx, 0c5h
64c4baa0: ed              in     eax, dx
64c4baa1: 6bd2c4          imul   edx, edx, 0c4h
64c4baa4: e27d            loop   7ff764c4bb23
64c4baa6: 02c2            add    al, dl
64c4baa8: c4              db     0c4h
64c4baa9: e37d            jecxz7dh
64c4baab: 39c1            cmp    ecx, eax
64c4baad: 01c5            add    ebp, eax
64c4baaf: f9              stc   
64c4bab0: fec1            inc    cl
64c4bab2: c4              db     0c4h
64c4bab3: c1797f03        sar    dword ptr [rcx+7fh], 03h
64c4bab7: c5              db     0c5h
64c4bab8: f8              clc   
64c4bab9: 28742408        sub    [rsp+08h], dh
64c4babd: c5              db     0c5h
64c4babe: f8              clc   
64c4babf: 77c3            ja     7ff764c4ba84
64c4bac1: 0f              db     0fh
64c4bac2: 1f              db     1fh
64c4bac3: 8400            test   [rax], al
64c4bac5: 0000            add    [rax], al
64c4bac7: 0000            add    [rax], al
64c4bac9: 0f              db     0fh
64c4baca: 1f              db     1fh
64c4bacb: 800000          add    byte ptr [rax], 00h
64c4bace: 0000            add    [rax], al
64c4bad0: c5              db     0c5h
64c4bad1: f8              clc   
64c4bad2: 29742408        sub    [rsp+08h], esi
64c4bad6: c5              db     0c5h
64c4bad7: f8              clc   
64c4bad8: 297c2418        sub    [rsp+18h], edi
64c4badc: 4c8b542428      mov    r10, [rsp+28h]
64c4bae1: 4c8b5c2430      mov    r11, [rsp+30h]
64c4bae6: 4b8d045b        lea    rax, [r11+r11*2]
64c4baea: c4              db     0c4h
64c4baeb: e27d            loop   7ff764c4bb6a
64c4baed: 5a              pop    edx
64c4baee: 21c4            and    esp, eax
64c4baf0: e27d            loop   7ff764c4bb6f
64c4baf2: 5a              pop    edx
64c4baf3: 6910c5fa6f02    imul   edx, [rax], 026ffac5
64c4baf9: c4              db     0c4h
64c4bafa: c17a6f08        sar    dword ptr [rdx+6fh], 08h
64c4bafe: c4              db     0c4h
64c4baff: a17a6f141a      mov    eax, [1a146f7a]
64c4bb04: c4              db     0c4h
64c4bb05: 817a6f1c18c4c3  cmp    dword ptr [rdx+6fh], c3c4181c
64c4bb0c: 7d38            jge    7ff764c4bb46
64c4bb0e: 0101            add    [rcx], eax
64c4bb10: c4              db     0c4h      <-- FAULT
64c4bb11: c3              ret   
64c4bb12: 7538            jnz    7ff764c4bb4c
64c4bb14: 0a01            or     al, [rcx]
64c4bb16: c4              db     0c4h
64c4bb17: 836d3814        sub    dword ptr [rbp+38h], 14h
64c4bb1b: 1901            sbb    [rcx], eax
64c4bb1d: c4              db     0c4h
64c4bb1e: 8365381c        and    dword ptr [rbp+38h], 1ch
64c4bb22: 1a01            sbb    al, [rcx]
64c4bb24: c5              db     0c5h
64c4bb25: fd              std   
64c4bb26: f6c4c5          test   ah, 0c5h
64c4bb29: f5              cmc   
64c4bb2a: f6              db     0f6h
64c4bb2b: cc              int    3
64c4bb2c: c5              db     0c5h
64c4bb2d: ed              in     eax, dx
64c4bb2e: f6d5            not    ch
64c4bb30: c5              db     0c5h
64c4bb31: e5f6            in     eax, 0f6h
64c4bb33: ddc5            ffree  st(5)
64c4bb35: fd              std   
64c4bb36: fd              std   
64c4bb37: c2c5f5          ret    f5c5
64c4bb3a: fd              std   
64c4bb3b: cb              retf   
64c4bb3c: c4              db     0c4h
64c4bb3d: e27d            loop   7ff764c4bbbc
64c4bb3f: 5a              pop    edx
64c4bb40: 7120            jno    7ff764c4bb62
64c4bb42: c4              db     0c4h
64c4bb43: e27d            loop   7ff764c4bbc2
64c4bb45: 5a              pop    edx
64c4bb46: 7930            jns    7ff764c4bb78
64c4bb48: c4              db     0c4h
64c4bb49: a17a6f145a      mov    eax, [5a146f7a]
64c4bb4e: c4              db     0c4h
64c4bb4f: 817a6f1c58c5fa  cmp    dword ptr [rdx+6fh], fac5581c
64c4bb56: 6f              outsd 
64c4bb57: 2402            and    al, 02h
64c4bb59: c4              db     0c4h
64c4bb5a: c17a6f2c        sar    dword ptr [rdx+6fh], 2ch
64c4bb5e: 00c4            add    ah, al
64c4bb60: 836d3814        sub    dword ptr [rbp+38h], 14h
64c4bb64: 59              pop    ecx
64c4bb65: 01c4            add    esp, eax
64c4bb67: 8365381c        and    dword ptr [rbp+38h], 1ch
64c4bb6b: 5a              pop    edx
64c4bb6c: 01c4            add    esp, eax
64c4bb6e: c3              ret   
64c4bb6f: 5d              pop    ebp
64c4bb70: 382401          cmp    [rcx+rax], ah
64c4bb73: 01c4            add    esp, eax
64c4bb75: c3              ret   
64c4bb76: 55              push   ebp
64c4bb77: 382c02          cmp    [rdx+rax], ch
64c4bb7a: 01c5            add    ebp, eax
64c4bb7c: ed              in     eax, dx
64c4bb7d: f6d6            not    dh
64c4bb7f: c5              db     0c5h

Built on R-COMP on Sun May 26 18:00:29 2024 using compiler version 1939

Windows 10 x64
Memory status: virtual free 134163245M/134217728M, commit limit 37582M, physical total 32718M

RAX =             11a0
RBX =      19217ecbbcf
RCX =      19126f8ef40
RDX =      19217ecb00b
RSI = ffffffffffffffff
RDI =         ffffffff
RBP =      19126f88b00
R8  =      19217ecb013
R9  =      19217ecb5eb
R10 =      19317ecb00e
R11 =              5e0
R12 = fffffffffffffffc
R13 =                4
R14 =              bc0
R15 =      19217ecc1b1
RSP =       4dc50f6e68
RIP =         64c4bb10
EFLAGS = 00010216


Crash reason: Access Violation

Crash context:
An out-of-bounds memory access (access violation) occurred in module 'x264-8'...
...reading address 0000019317ECB00E.

Pointer dumps:

RBX   19217ecbbcb: b7b7b6b5 b7b7b6b7 b7b7b7b7 b9b8b7b7 bcbcbbba b6b8bdbd bab9b8b6 bbbbbbbb
RCX   19126f8ef40: b6b7b8ba b3b4b5b6 b5b4b3b3 babab8b7 b8b8b9ba b6b7b8b8 b7b6b5b6 bcbcbbb9
RDX   19217ecb00b: b7b8b7b6 b3b4b5b6 b4b2b1b1 b8b6b5b5 b8b7b6b7 b5b7bab9 b9b9b8b6 bbbbbaba
RSP   4dc50f6e68: 64be5700 00000000 00000000 00000000 00000000 00000000 00000000 00000000
      4dc50f6e88: 00000000 00000000 17ecb00e 00000193 000005e0 00000000 c50f70e0 0000004d
      4dc50f6ea8: 00000010 00000000 64c84500 00000000 ffffffff ffffffff 26f8ef40 00000191
      4dc50f6ec8: 000005e0 00000000 ffffffff 0000004d 17ecaa30 00000192 26f22280 00000191
RBP   19126f88b00: 0010f3ff 00000012 00000003 00000000 00000001 00000000 00000009 000005a0
      19126f88b20: 00000438 00000002 00000008 00000032 000244d2 00000000 00000001 00000001
      19126f88b40: 00000000 00000005 00000000 00000002 00000002 00000001 00000000 00000010
      19126f88b60: 00000001 000000fa 00000019 00000028 00000000 00000008 00000002 00000000
R8    19217ecb013: b4b2b1b1 b8b6b5b5 b8b7b6b7 b5b7bab9 b9b9b8b6 bbbbbaba bababbbb babababb
R9    19217ecb5eb: b7b8b7b5 b5b6b6b7 b5b4b3b4 b7b7b7b6 babab8b7 b5b7bbba bab9b7b5 bbbbbabb
R15   19217ecc1b1: b9b9b8b7 bababab9 bdbdbcbb bdbebdbd b8b7b8ba bbbabab9 bdbcbbbb bcbcbcbd

Thread call stack:
64c4bb10: x264-8!VDDriverProc [64b40000+9290+102880]
64bcf6a2: x264-8!VDDriverProc [64b40000+9290+86412]
64c3f593: x264-8!VDDriverProc [64b40000+9290+f6303]
64b67b49: x264-8!VDDriverProc [64b40000+9290+1e8b9]
64bbe5b6: x264-8!VDDriverProc [64b40000+9290+75326]
64c3ffb5: x264-8!VDDriverProc [64b40000+9290+f6d25]
64bd7781: x264-8!VDDriverProc [64b40000+9290+8e4f1]
64c1e86a: x264-8!VDDriverProc [64b40000+9290+d55da]
64c1d9b4: x264-8!VDDriverProc [64b40000+9290+d4724]
64b8acae: x264-8!VDDriverProc [64b40000+9290+41a1e]
64c1a440: x264-8!VDDriverProc [64b40000+9290+d11b0]
64c01cc7: x264-8!VDDriverProc [64b40000+9290+b8a37]
64c236f1: x264-8!VDDriverProc [64b40000+9290+da461]
64c09b1c: x264-8!VDDriverProc [64b40000+9290+c088c]
64bd37e6: x264-8!VDDriverProc [64b40000+9290+8a556]
64b58a6f: x264-8!VDDriverProc [64b40000+9290+f7df]
7ffd16b4cb51: ntdll!RtlFindActivationContextSectionString [7ffd16b30000+1c6c0+491]
7ffd16b4d23b: ntdll!RtlFindActivationContextSectionGuid [7ffd16b30000+1cd80+4bb]
7ffd16b4c7bc: ntdll!RtlFindActivationContextSectionString [7ffd16b30000+1c6c0+fc]
7ffd16b4c3c8: ntdll!RtlInitUnicodeStringEx [7ffd16b30000+1c270+158]
7ffcdc2e4517: FileSyncShell64!?SizeUnknown@QoS@@YAIXZ [7ffcdc2e0000+43e0+137]
7ffd16b4aced: ntdll!RtlDosApplyFileIsolationRedirection_Ustr [7ffd16b30000+1a9e0+30d]
7ffd16b51374: ntdll!RtlEqualUnicodeString [7ffd16b30000+20d10+664]
7ffd16b91953: ntdll!RtlDetectHeapLeaks [7ffd16b30000+61770+1e3]
7ffd16b49877: ntdll!RtlDeactivateActivationContextUnsafeFast [7ffd16b30000+197b0+c7]
7ffd16b4f5f7: ntdll!RtlGetFullPathName_UstrEx [7ffd16b30000+1d890+1d67]
7ffd16b4fcf4: ntdll!RtlGetFullPathName_UstrEx [7ffd16b30000+1d890+2464]
7ffd16b46dad: ntdll!RtlDosPathNameToNtPathName_U_WithStatus [7ffd16b30000+16be0+1cd]
7ffd16b46cdf: ntdll!RtlDosPathNameToNtPathName_U_WithStatus [7ffd16b30000+16be0+ff]
7ffd16b46d4b: ntdll!RtlDosPathNameToNtPathName_U_WithStatus [7ffd16b30000+16be0+16b]
7ffd16b5121f: ntdll!RtlEqualUnicodeString [7ffd16b30000+20d10+50f]
7ffd16b5b44d: ntdll!RtlAllocateHeap [7ffd16b30000+2a9a0+aad]
7ffd16b4a8fc: ntdll!RtlReleaseRelativeName [7ffd16b30000+1a320+5dc]
7ffd16b4a407: ntdll!RtlReleaseRelativeName [7ffd16b30000+1a320+e7]
7ffd16b5df6f: ntdll!RtlAllocateHeap [7ffd16b30000+2a9a0+35cf]
7ffd16b4f47e: ntdll!RtlGetFullPathName_UstrEx [7ffd16b30000+1d890+1bee]
7ffd16b50b31: ntdll!RtlTryEnterCriticalSection [7ffd16b30000+207e0+351]
7ffd16b5b90b: ntdll!RtlAllocateHeap [7ffd16b30000+2a9a0+f6b]
7ffd16b5b3c7: ntdll!RtlAllocateHeap [7ffd16b30000+2a9a0+a27]
7ffd16b602c9: ntdll!LdrGetProcedureAddressForCaller [7ffd16b30000+2fdc0+509]
7ffcdc3882b4: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+4cd54]
7ffcdc387ff6: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+4ca96]
7ffd0fe029ff: WTSAPI32!IsInteractiveUserSession [7ffd0fe00000+2630+3cf]
7ffcdc388531: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+4cfd1]
7ffcdc387d49: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+4c7e9]
7ffcdc36a7c6: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+2f266]
7ffcdc36b762: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+30202]
7ffd16b49877: ntdll!RtlDeactivateActivationContextUnsafeFast [7ffd16b30000+197b0+c7]
7ffcdc36b16d: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+2fc0d]
7ffd16b49a26: ntdll!RtlActivateActivationContextUnsafeFast [7ffd16b30000+19900+126]
7ffd16b49993: ntdll!RtlActivateActivationContextUnsafeFast [7ffd16b30000+19900+93]
7ffcdc36b3bb: FileSyncShell64!?GetLogObfuscationKeyManger@@YAJPEAPEAVILogObfuscationKeyManager@@@Z [7ffcdc2e0000+5b560+2fe5b]
7ffd16b49aff: ntdll!RtlActivateActivationContextUnsafeFast [7ffd16b30000+19900+1ff]
7ffd16b6e693: ntdll!RtlGetVersion [7ffd16b30000+3e4e0+1b3]
7ffd16b8eec4: ntdll!LdrEnumerateLoadedModules [7ffd16b30000+5edd0+f4]
7ffd16b47955: ntdll!LdrShutdownThread [7ffd16b30000+17450+505]
7ffd16b47877: ntdll!LdrShutdownThread [7ffd16b30000+17450+427]
7ffd16b478aa: ntdll!LdrShutdownThread [7ffd16b30000+17450+45a]
7ffd16b4791d: ntdll!LdrShutdownThread [7ffd16b30000+17450+4cd]
7ffd16b4793d: ntdll!LdrShutdownThread [7ffd16b30000+17450+4ed]
7ffd16ba60bd: ntdll!LdrInitializeThunk [7ffd16b30000+75c20+49d]
7ffd16ba5c83: ntdll!LdrInitializeThunk [7ffd16b30000+75c20+63]
7ffd16ba5c38: ntdll!LdrInitializeThunk [7ffd16b30000+75c20+18]
7ffd16b94821: ntdll!RtlWakeAddressAll [7ffd16b30000+646e0+141]
7ffd16b5fae2: ntdll!RtlEnterCriticalSection [7ffd16b30000+2faa0+42]
7ffd16b81910: ntdll!RtlRestoreLastWin32Error [7ffd16b30000+518d0+40]
7ffd16b95897: ntdll!RtlSleepConditionVariableCS [7ffd16b30000+65770+127]
7ffd143b478d: KERNELBASE!OpenThreadToken [7ffd14380000+34740+4d]
64b57416: x264-8!VDDriverProc [64b40000+9290+e186]
64b5b0d0: x264-8!VDDriverProc [64b40000+9290+11e40]
64b51b88: x264-8!VDDriverProc [64b40000+9290+88f8]
64b5195b: x264-8!VDDriverProc [64b40000+9290+86cb]
64b51a0a: x264-8!VDDriverProc [64b40000+9290+877a]
64b65f00: x264-8!VDDriverProc [64b40000+9290+1cc70]
7ffd1524aa7a: msvcrt!__threadid [7ffd15210000+3a800+27a]
64b51813: x264-8!VDDriverProc [64b40000+9290+8583]
7ffd1524af5a: msvcrt!_beginthreadex [7ffd15210000+3ae30+12a]
7ffd1524b02c: msvcrt!_endthreadex [7ffd15210000+3af80+ac]
7ffd15167344: KERNEL32!BaseThreadInitThunk [7ffd15150000+17330+14]
7ffd16b7cc91: ntdll!RtlUserThreadStart [7ffd16b30000+4cc70+21]

-- End of report

avs script:

SetFilterMTMode("qtgmc", 2)
FFmpegSource2("B1_t00.mkv", atrack=-1)
AssumeBFF()
QTGMC(Preset="slower", EdiThreads=3)
LanczosResize(1440,1080)
KNLMeansCL(d=1, a=2, s=4, h=1.2, wref=1.0, channels="auto", wmode=0, device_type="auto", mode_9_to_15bits=1, info=false)
Prefetch(10)

Edit to add: I had converted several files early on successfully and then this started happening and have 1 successful encode on it since I started getting the errors.

[Selur]

Sorry, I have no clue about VirtualDub and the x264 vfw component it uses.
Seeing that the out-of-memory occurs in the vfw x264, adjusting the memory usage in for Avisynth seems like the wrong approach.
Things I would recommend trying:

• use AVSMeter to seem whether Avisynths memory consumption is the problem. (assuming your input video is SD, I doubt it)
  If the memory issue stems from Avisynth, using a different source filter and other adjustments to the script might help.
  
• try using VirtualDubs external encoder with x264s cli encoder instead of the vfw interface to see whether it is a general problem in x264 or just x264s vfw interface. If it's the vfw interface, either don't use it or contact the developer. Additionally, deactivating different ASM optimizations, assuming the vfw interface allows it, might help.
  

Since you mentioned Hybrid, if you can provide proper details ([INFO] Infos needed to fix&reproduce bugs,..) and a source sample which allows reproducing the problem, I can look into that.

Cu Selur