Selur's Little Message Board

Full Version: VapourSynth crashing on large files
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hi there,

I'm scratching my head trying to get Vapoursynth working on some VHS captures I did (FFV1 compressed, 4:2:2 8-bit) and encoding with x264.
Basically small snippets work, even with Hybrid's *experimental frame cut function. But when I try to encode the whole file, the process crashes, sometimes almost instantly, sometimes after several minutes of encoding. I somehow managed to get one 30min video through on the previous 2025.10.15 development build (juggling with deinterlacing settings, adding or removing various Vapoursynth filters (notably CCD), enabling Bestsource (Cuda), adding audio or not, framecut etc), but now I'm again stuck on another long video, while smaller segments do get encoded.

p.s. I attached a debug file, and I'm not sure what's exactly recorded in there, but while debugging was enabled there was a crash of a large file while afterwards a small segment (cut out with VirtualDub from the same video) did get encoded.
The used Vapoursynth script:
Code:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
import ctypes
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("C:/Program Files/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll")
# loading plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SharpenFilter/AWarpSharp2/libawarpsharp2.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/Cnr2/libcnr2.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/FFT3DFilter/fft3dfilter.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/FluxSmooth/libfluxsmooth.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DeCrawlFilter/DeDot/libdedot.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/GrainFilter/AddGrain/AddGrain.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/EEDI3m.dll")# vsQTGMC
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/ResizeFilter/nnedi3/vsznedi3.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DeinterlaceFilter/Bwdif/Bwdif.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/libllvmexpr.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/ZSmooth/zsmooth.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DerainbowFilter/DeCross/libdecross.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/BestSource/BestSource.dll")
# Import scripts
import dehalo
import color
import artifacts
import qtgmc
import validate
# Source: 'D:\FABIO157-elvinRavenna[PANA-TBC3000-ATI600].avi'
# Current color space: YUV422P8, bit depth: 8, resolution: 720x576, frame rate: 25fps, scanorder: top field first, yuv luminance scale: limited, matrix: 470bg, format: FFV1
# Loading D:\FABIO157-elvinRavenna[PANA-TBC3000-ATI600].avi using BestSource)
clip = core.bs.VideoSource(source="D:/FABIO157-elvinRavenna[PANA-TBC3000-ATI600].avi", cachepath="D:/tempHybrid/FABIO157-elvinRavenna[PANA-TBC3000-ATI600]_bestSource_2025-11-15@17_48_15_4910", track=0, hwdevice="cuda")
frame = clip.get_frame(0)
# setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT470_BG)
# setting color transfer (vs.TRANSFER_BT601), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT601)
# setting color primaries info (to vs.PRIMARIES_BT470_BG), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT470_BG)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED)
# making sure frame rate is set to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# making sure the detected scan type is set (detected: top field first)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_TOP) # tff
# rainbow removal using DeCross
clip = core.decross.DeCross(clip)
# Deinterlacing using QTGMC
clip = qtgmc.QTGMC(clip, Preset="Fast", TFF=True)# new fps: 50
# Making sure content is preceived as frame based
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive
clip = core.dedot.Dedot(clip)
# adjusting color using Tweak
clip = color.Tweak(clip, hue=-0.02, sat=0.99, cont=0.99, bright=4, coring=False)
clip = core.std.Crop(clip, left=18, right=2, top=0, bottom=10)# cropping to 700x566
# denoising using FluxSmooth
clip = core.flux.SmoothT(clip)
# denoising using FFT3DFilter
clip = core.fft3dfilter.FFT3DFilter(clip)
# chroma denoising using VsCnr2
clip = core.cnr2.Cnr2(clip, mode="oxx")
# adjusting color space from YUV422P8 to RGB24 for vsCCD
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_in_s="limited", range_s="full")
# chroma denoising using CCD
clip = core.zsmooth.CCD(clip, threshold=10.00, temporal_radius=1, points=[True,True,False])
# adjusting color space from RGB24 to YUV444P16 for vsEdgeCleaner
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="470bg", range_in_s="full", range_s="limited") # additional resize to allow target color sampling
# Dehalo using EdgeCleaner
clip = dehalo.EdgeCleaner(clip)
# letterboxing 700x566 to 720x576
clip = core.std.AddBorders(clip, left=10, right=10, top=5, bottom=5)
# adjusting output color from: YUV444P16 to YUV422P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV422P8, dither_type="error_diffusion")
# set output frame rate to 50fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=50, fpsden=1)
# output
clip.set_output()
seems fine, encoding does start, but crashes after 197 frames:
Code:
2025.11.15 - 17:50:09_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: 2025-11-15@17_48_15_4910_02_video ### D:\FABIO157-elvinRavenna[PANA-TBC3000-ATI600]_new_new.mkv ### x264 [info]: using SAR=16/15
2025.11.15 - 17:50:09_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: 2025-11-15@17_48_15_4910_02_video ### D:\FABIO157-elvinRavenna[PANA-TBC3000-ATI600]_new_new.mkv ### x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
2025.11.15 - 17:50:09_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: 2025-11-15@17_48_15_4910_02_video ### D:\FABIO157-elvinRavenna[PANA-TBC3000-ATI600]_new_new.mkv ### x264 [info]: profile High 4:2:2, level 5.1, 4:2:2, 8-bit
2025.11.15 - 17:50:10_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 6 frames: 1.82 fps, 7419.80 kb/s
2025.11.15 - 17:50:10_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 11 frames: 3.07 fps, 6009.38 kb/s
2025.11.15 - 17:50:10_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 16 frames: 4.15 fps, 5575.98 kb/s
2025.11.15 - 17:50:10_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 21 frames: 5.09 fps, 5362.55 kb/s
2025.11.15 - 17:50:11_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 26 frames: 5.92 fps, 5221.49 kb/s
2025.11.15 - 17:50:11_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 31 frames: 6.66 fps, 4992.08 kb/s
2025.11.15 - 17:50:11_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 36 frames: 7.33 fps, 4702.92 kb/s

2025.11.15 - 17:50:20_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: x264 output: 197 frames: 14.33 fps, 4092.44 kb/s
2025.11.15 - 17:50:20_Windows 10 Version 22H2 (64bit)_2025.11.09.1 - level 9: 2025-11-15@17_48_15_4910_02_video ### D:\FABIO157-elvinRavenna[PANA-TBC3000-ATI600]_new_new.mkv ### Helper(1) finished with exitCode: -1073741819 and exitStatus:
Could be a memory or cpu or heat problem?
Is your system over- or underclocked?
Are you running out of space?
Could be some other software interfering or a driver problem.
Since FFV1 is not supported by cuda, try whether not using hwdevice="cuda" or using another source filter makes a difference.

Cu Selur