VapourSynth crashing on large files

[OhNoxius]

Hi there,

I'm scratching my head trying to get Vapoursynth working on some VHS captures I did (FFV1 compressed, 4:2:2 8-bit) and encoding with x264.
Basically small snippets work, even with Hybrid's *experimental frame cut function. But when I try to encode the whole file, the process crashes, sometimes almost instantly, sometimes after several minutes of encoding. I somehow managed to get one 30min video through on the previous 2025.10.15 development build (juggling with deinterlacing settings, adding or removing various Vapoursynth filters (notably CCD), enabling Bestsource (Cuda), adding audio or not, framecut etc), but now I'm again stuck on another long video, while smaller segments do get encoded.

p.s. I attached a debug file, and I'm not sure what's exactly recorded in there, but while debugging was enabled there was a crash of a large file while afterwards a small segment (cut out with VirtualDub from the same video) did get encoded.

[Selur]

The used Vapoursynth script:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
import ctypes
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("C:/Program Files/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll")
# loading plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SharpenFilter/AWarpSharp2/libawarpsharp2.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/Cnr2/libcnr2.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/FFT3DFilter/fft3dfilter.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/FluxSmooth/libfluxsmooth.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DeCrawlFilter/DeDot/libdedot.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/GrainFilter/AddGrain/AddGrain.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/EEDI3m.dll")# vsQTGMC
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/ResizeFilter/nnedi3/vsznedi3.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DeinterlaceFilter/Bwdif/Bwdif.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/libllvmexpr.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/ZSmooth/zsmooth.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DerainbowFilter/DeCross/libdecross.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/BestSource/BestSource.dll")
# Import scripts
import dehalo
import color
import artifacts
import qtgmc
import validate
# Source: 'D:\FABIO157-elvinRavenna[PANA-TBC3000-ATI600].avi'
# Current color space: YUV422P8, bit depth: 8, resolution: 720x576, frame rate: 25fps, scanorder: top field first, yuv luminance scale: limited, matrix: 470bg, format: FFV1
# Loading D:\FABIO157-elvinRavenna[PANA-TBC3000-ATI600].avi using BestSource)
clip = core.bs.VideoSource(source="D:/FABIO157-elvinRavenna[PANA-TBC3000-ATI600].avi", cachepath="D:/tempHybrid/FABIO157-elvinRavenna[PANA-TBC3000-ATI600]_bestSource_2025-11-15@17_48_15_4910", track=0, hwdevice="cuda")
frame = clip.get_frame(0)
# setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT470_BG)
# setting color transfer (vs.TRANSFER_BT601), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT601)
# setting color primaries info (to vs.PRIMARIES_BT470_BG), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT470_BG)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED)
# making sure frame rate is set to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# making sure the detected scan type is set (detected: top field first)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_TOP) # tff
# rainbow removal using DeCross
clip = core.decross.DeCross(clip)
# Deinterlacing using QTGMC
clip = qtgmc.QTGMC(clip, Preset="Fast", TFF=True)# new fps: 50
# Making sure content is preceived as frame based
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive
clip = core.dedot.Dedot(clip)
# adjusting color using Tweak
clip = color.Tweak(clip, hue=-0.02, sat=0.99, cont=0.99, bright=4, coring=False)
clip = core.std.Crop(clip, left=18, right=2, top=0, bottom=10)# cropping to 700x566
# denoising using FluxSmooth
clip = core.flux.SmoothT(clip)
# denoising using FFT3DFilter
clip = core.fft3dfilter.FFT3DFilter(clip)
# chroma denoising using VsCnr2
clip = core.cnr2.Cnr2(clip, mode="oxx")
# adjusting color space from YUV422P8 to RGB24 for vsCCD
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_in_s="limited", range_s="full")
# chroma denoising using CCD
clip = core.zsmooth.CCD(clip, threshold=10.00, temporal_radius=1, points=[True,True,False])
# adjusting color space from RGB24 to YUV444P16 for vsEdgeCleaner
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="470bg", range_in_s="full", range_s="limited") # additional resize to allow target color sampling
# Dehalo using EdgeCleaner
clip = dehalo.EdgeCleaner(clip)
# letterboxing 700x566 to 720x576
clip = core.std.AddBorders(clip, left=10, right=10, top=5, bottom=5)
# adjusting output color from: YUV444P16 to YUV422P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV422P8, dither_type="error_diffusion")
# set output frame rate to 50fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=50, fpsden=1)
# output
clip.set_output()

seems fine, encoding does start, but crashes after 197 frames:

2025.11.15 - 17:50:09_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: 2025-11-15@17_48_15_4910_02_video ### D:\FABIO157-elvinRavenna[PANA-TBC3000-ATI600]_new_new.mkv ### x264 [info]: using SAR=16/15
2025.11.15 - 17:50:09_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: 2025-11-15@17_48_15_4910_02_video ### D:\FABIO157-elvinRavenna[PANA-TBC3000-ATI600]_new_new.mkv ### x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
2025.11.15 - 17:50:09_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: 2025-11-15@17_48_15_4910_02_video ### D:\FABIO157-elvinRavenna[PANA-TBC3000-ATI600]_new_new.mkv ### x264 [info]: profile High 4:2:2, level 5.1, 4:2:2, 8-bit
2025.11.15 - 17:50:10_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 6 frames: 1.82 fps, 7419.80 kb/s
2025.11.15 - 17:50:10_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 11 frames: 3.07 fps, 6009.38 kb/s
2025.11.15 - 17:50:10_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 16 frames: 4.15 fps, 5575.98 kb/s
2025.11.15 - 17:50:10_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 21 frames: 5.09 fps, 5362.55 kb/s
2025.11.15 - 17:50:11_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 26 frames: 5.92 fps, 5221.49 kb/s
2025.11.15 - 17:50:11_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 31 frames: 6.66 fps, 4992.08 kb/s
2025.11.15 - 17:50:11_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 36 frames: 7.33 fps, 4702.92 kb/s

2025.11.15 - 17:50:20_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 197 frames: 14.33 fps, 4092.44 kb/s
2025.11.15 - 17:50:20_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: 2025-11-15@17_48_15_4910_02_video ### D:\FABIO157-elvinRavenna[PANA-TBC3000-ATI600]_new_new.mkv ### Helper(1) finished with exitCode: -1073741819 and exitStatus:

Could be a memory or cpu or heat problem?
Is your system over- or underclocked?
Are you running out of space?
Could be some other software interfering or a driver problem.
Since FFV1 is not supported by cuda, try whether not using hwdevice="cuda" or using another source filter makes a difference.

Cu Selur

[OhNoxius]

Thanks for the suggestions.

I'm working on a i7-4771 // 8GB DDR3 // GTX960 2GB, nothing overclocked, C: has 34GB and D: has 50GB free space.

Setting bestsource=cpu didn't help, nor using FFMpegSource2 or LWLibAVSource. I remember setting bestsource=cuda because that somehow caused some succesfull encodes (although that shouldn't make any sense...)
Maybe this is relevant: I remember cropping the previous 30min beforehand and applying the CCD filter in VirtualDub2, before applying VapourSynth filters and encoding with Hybrid (cropping & letterboxing disabled).

Went to an i5-9600KF // 16GB // GTX1070ti 8GB and tried encoding the same video with same VapourSynth filters on. Crashed again, almost instantly. Then set bestsource=cuda & disabled VapourSynth CCD, did a VapourSynth preview (compare view) do be sure the VapourSynth was working, and managed to get it encoded untill around 50%... then crash.
Then set bestsource=cpu and x264 profile=High (instead of High4:2:2 with 8-bit precision), and got it to 100%!
-> enabled audio encoding to FLAC: success
-> enabled CCD: crash
-> disabled CCD, enabled cut support (frame selection): crash

Went back to my other machine and tried to repeat the same actions... and got to a 100%! (no audio or CCD enabled. Tried enabling cut support and encoding did start but crashed underway)

All this actually feels so random... I could swear I had tried the exact same settings on my i7 before, and now it suddenly works? Is there any way to know where in the process the crash occurs? Is it a x264 thing, or is Vapoursynth crashing?

Thanks
Karel

p.s.  I have another i9-13900H laptop (with no dedicated graphics card) I can test on...

[Selur]

Based in the debug output the Decoder crashes.
So it is probably a filter, the source or something in the system and not the encoder.