20.12.2022, 06:27 (This post was last modified: 20.12.2022, 06:30 by Selur.)
Since the file gets analyzed here without issues.
a. create a debug output of the analysis (read sticky on how-to) and share it.
b. try whether moving the source into a folder with only ASCII characters help. (save for output&temp folder)
If you already got an error during the analysis, I'm not surprised there are issues with the decoding of the source.
Cu Selur
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
I bumped into similar issue, randomly getting this error with some x264 jobs:
Helper(1) finished with exitCode: -1073741819 and exitStatus: 1
-> 2024-08-15@21_47_56_3910_06_video crashed: Helper1, exitCode: -1073741819
Aborting '2024-08-15@21_47_56_3910_06_video' due to: Helper1, exitCode: -1073741819
Attached is my HybridDebugOutput.txt but I cannot see much hints in it as to why it crashes. Usually I'm able to simply reset and restart a job to make it finish successfully but this one has proven to be a tough one (this was my third retry, have not tried rebooting the whole machine yet though). I was at first suspecting laptop sleep function first but even leaving Hybrid open, disabling sleep/hibernation and forcing the machine "active" using PowerToys "Keep awake" (with Keep screen on) did not seem to help. Previously I have been able to encode this same file with slightly different settings.
PS.: And maybe a feature request worth thinking, related to this kind of issues (unless there already is something like it?):
"Retry CRASHED jobs X times" and/or "Continue to other WAITING jobs after a job has CRASHED", as for some reason the other jobs in the list remain WAITING.
16.08.2024, 14:25 (This post was last modified: 16.08.2024, 14:27 by Selur.)
Quote:PS.: And maybe a feature request worth thinking, related to this kind of issues (unless there already is something like it?):
"Retry CRASHED jobs X times" and/or "Continue to other WAITING jobs after a job has CRASHED", as for some reason the other jobs in the list remain WAITING.
The number of parallel jobs is 1, so if that one crashes, Hybrid stops.
"Retry CRASHED jobs X times" that sounds like insanity.
"Continue to other WAITING jobs after a job has CRASHED", that could make sense, but since a crash often has a reason that could potentially influence other jobs too, I'm not going to implement it.
Looking at the debug output:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
import ctypes
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("C:/Program Files/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll")
# loading plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/HQDN3D/libhqdn3d.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/GrainFilter/RemoveGrain/RemoveGrainVS.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/GrainFilter/AddGrain/AddGrain.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/EEDI3m_opencl.dll")# vsQTGMC
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/ResizeFilter/nnedi3/NNEDI3CL.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/scenechange.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DeinterlaceFilter/Bwdif/Bwdif.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/d2vSource/DGDecode.dll")
# Import scripts
import havsfunc
import validate
# Source: 'C:\Users\hjatsu\Desktop\vhs\VHSREC2#001.vob'
# Current color space: YUV420P8, bit depth: 8, resolution: 720x576, frame rate: 25fps, scanorder: top field first, yuv luminance scale: limited, matrix: 470bg, transfer: bt.470 system b/g, primaries: bt.601 pal
# Loading C:\Users\hjatsu\Desktop\vhs\VHSREC2#001.vob using DGDecode
clip = core.dgdecode.MPEG2Source("C:/Users/hjatsu/AppData/Local/Temp/2024-08-15@21_47_56_3910.d2v",info=3)# 25 fps, scanorder: top field first
frame = clip.get_frame(0)
# setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT470_BG)
# setting color transfer (vs.TRANSFER_BT470_BG), if it is not set.
if validate.transferIsInvalid(clip):
clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT470_BG)
# setting color primaries info (to vs.PRIMARIES_BT470_BG), if it is not set.
if validate.primariesIsInvalid(clip):
clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT470_BG)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED)
# making sure frame rate is set to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# making sure the detected scan type is set (detected: top field first)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_TOP) # tff
# cutting from frame 0 to 27976 - WARNING: This might cause synch issues
clip = core.std.Trim(clip=clip, first=0, last=27976)
# Deinterlacing using QTGMC
clip = havsfunc.QTGMC(Input=clip, Preset="Very Slow", TFF=True, opencl=True, device=0) # new fps: 50
# Making sure content is preceived as frame based
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive
clip = core.std.CropRel(clip=clip, left=16, right=12, top=2, bottom=10)# cropping to 692x564
# denoising using HQDN3D
clip = core.hqdn3d.Hqdn3d(clip=clip, lum_spac=2.00, chrom_spac=2.00, lum_tmp=4.00, chrom_tmp=6.00)
# set output frame rate to 50fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=50, fpsden=1)
# output
clip.set_output()
looks fine to me, source is trimmed to 27977 frames and boobed, so the output should be 55954 frames.
The decoder call:
But then the decoder crashes " Helper(1) finished with exitCode: -1073741819 and exitStatus: 1"
-1073741819 indicates an access violation in the helper, so somewhere in the Vapoursynth processing.
This is usually caused by one of these things:
a glitch in the source => check whether you can navigate in the Vapoursynth Preview around frame 55698.
heat issues
memory issues
some driver issues (unlikely since it worked for 55698 frames)
overclocking of cpu/gpu/ram (yes, even when every other test seems fine; encoding and intense filtering is really sensitive)
a bug in a filter (unlikely since it worked for 55698 frames)
some antivirus tool (or the os) interfering (especially mcafee and nortons live scanning stuff can cause problems)
runnning out of ram (there isn't always an out-of-memory error is memory gets spars)
Cu Selur
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
Thanks for the ideas! Did investigate them but so far unsuccessfully, will try and test some more.
(16.08.2024, 14:25)Selur Wrote: "Retry CRASHED jobs X times" that sounds like insanity.
Very much agree but this was what got me through this very same batch the first time, just resetting and retrying until there was no crash
(16.08.2024, 14:25)Selur Wrote: But then the decoder crashes " Helper(1) finished with exitCode: -1073741819 and exitStatus: 1"
-1073741819 indicates an access violation in the helper, so somewhere in the Vapoursynth processing.
This is usually caused by one of these things:
a glitch in the source => check whether you can navigate in the Vapoursynth Preview around frame 55698.
heat issues
memory issues
some driver issues (unlikely since it worked for 55698 frames)
overclocking of cpu/gpu/ram (yes, even when every other test seems fine; encoding and intense filtering is really sensitive)
a bug in a filter (unlikely since it worked for 55698 frames)
some antivirus tool (or the os) interfering (especially mcafee and nortons live scanning stuff can cause problems)
runnning out of ram (there isn't always an out-of-memory error is memory gets spars)
a glitch in the source:
Was able to move around the frame without issues in the preview. Did some more tests with another source and saw that it crashed three times in exactly the same frame (12114) but also with that, no issues playing through the frame and onwards in Vapoursynth preview window. Before this I also happened to have two concecutive succesful finishes for the same clip (but noticed the debug setting had switched off in app restart.)
heat issues:
I was very hopeful with this one when I saw 100C peak CPU temperatures in HWMonitor (CPU at around 50%). Proceeded to set a --threads 2 for the job which lowered the temps to 70C (60C is my idle) but unfortunately it crashed at the first try, only slightly later at frame 12130.
(I have also made an observation that if I start a job but then move window focus to some other app than Hybrid, the x264 encode subjob runs at a very low speed and CPU usage, and only visiting the Hybrid window will put things into full throttle)
some driver issues:
Potentially possible but I also (accidentially) ran Vapoursynth in OpenCL mode with a wrong device ID (the Intel iGPU instead of Nvidia GPU) and it crashed on some runs and finished succesfully on another run, so maybe not that likely?
overclocking of cpu/gpu/ram:
Not overclocked, pretty new Dell Precision 5680 laptop (Core i9-13900H & RTX 3500).
a bug in a filter:
Cannot comment on this, but no issues seen in previews. But anecdotally, the crash/success -ratio seems much lower with the experimental Hybrid build I'm using now vs. the latest stable one I used (without GPU support) to encode the earlier batch.
some antivirus tool (or the os) interfering:
Added the Hybrid folder and the source/output folder to the AV (Withsecure) exceptions. Will still need to try some more with the AV app disabled totally but the first run after disabling it was a successful one, the second a crash.
running out of ram:
I have 32GB of ram and the maximum utilization I have seen in the Task Manager is 14GB (whole system).
I doubt this is at all helpful but here's a bit longer log file, with a couple of crashes and one successful run (and manual stops where I was investigating the CPU temps): https://wormhole.app/nZWAR#yXxOdg2tR91f9T8c9biwqQ
Perhaps I should try the "Prefer FFmpeg" Decoding setting next if you say that the error means that the decoder is crashing?
Another thing you could try: disabling the efficiency cores in the bios.
Iirc someone over at doom9s complained that he had instability during encoding&co unless he disabled them.
To figure out whether it could be related to some libraty, try whether it makes a difference if you use a faster preset with QTGMC and disable other filters.
Another thing: move your temp folder (Config->Paths) to a dedicated folder and exclude it from any virus scans.
Cu Selur
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
After this I could only see P-Cores in HWMonitor, and the clocks appear to remain very stable comparing to earlier tests. So far 4/5 jobs finished successfully! So still one crash, but much better ratio than earlier. Keeping my fingers crossed with the longer clips but looks promising...
If E-Cores are indeed the culprit then I guess it might be safe to re-enable TurboBoost to get the clocks back up from 2,5GHz to the ~5GHz level again. The temps appear to remain at almost idle levels now.
Thanks for the help getting into the (likely) root of this!
Will test the lighter filtering setup too with the next batch.
Unfortunately, bad news. For some reason the shorter clips had higher success rate but with longer ones it was back to about 50/50.
Did some digging in the logs an found out this pattern:
Crash at 18040/18280 (240 frames left) (98,68%)
Crash at 86538/86780 (242 frames left) (99,72%)
Crash at 87216/87458 (242 frames left) (99,72%)
Could it be a coincidence that I have "--sync-lookahead 120 --rc-lookahead 120" and the source is being bobbed, so 240 refers to source clip ending? Although, previously I had about the same 50/50 ratio of crashes with the very first batch which was using all default settings and veryfast 2-pass encoding, sometimes crashing the 1st pass, sometimes the 2nd pass.
(Note: I'm also using the *experimental* Cut Support to cut different parts of the material. Maybe next I will try to trim my VOB to a short MPG beforehand to eliminate that varible from the equation.)
But anyway, I created a test case which appeared to crash every time, though a couple of times with exitcode 0 instead of the usual one, cannot guess why.
The debug log and the job file are linked below as well as the source material, in case you might have an opportunity to test this out yourself sometime:
per(1) finished with exitCode: -1073741819 and exitStatus: 1
Decoder crashed after 616 frames.
3rd crash, line 5270, is the same as the first.
4th crash, line 6751, is the same as the first.
5th crash, line 7933, is the same as the second, crashed at the same frame.
6th crash, line 9101, is the same as the second, crashed at the same frame.
I tested a similar script with your source:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
import ctypes
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'F:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("F:/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll")
# loading plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/DenoiseFilter/HQDN3D/libhqdn3d.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/GrainFilter/RemoveGrain/RemoveGrainVS.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/GrainFilter/AddGrain/AddGrain.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/EEDI3m_opencl.dll")# vsQTGMC
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/nnedi3/NNEDI3CL.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/scenechange.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/DeinterlaceFilter/Bwdif/Bwdif.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/d2vSource/DGDecode.dll")
# Import scripts
import havsfunc
import validate
# Source: 'C:\Users\Selur\Desktop\VHSREC4#001.vob'
# Current color space: YUV420P8, bit depth: 8, resolution: 720x576, frame rate: 25fps, scanorder: top field first, yuv luminance scale: limited, matrix: 470bg, transfer: bt.470 system b/g, primaries: bt.601 pal
# Loading C:\Users\Selur\Desktop\VHSREC4#001.vob using DGDecode
clip = core.dgdecode.MPEG2Source("J:/tmp/vob_6fc6b21b9bfb55ae4e6e5f24686c4223_853323747.d2v",info=3)# 25 fps, scanorder: top field first
frame = clip.get_frame(0)
# setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT470_BG)
# setting color transfer (vs.TRANSFER_BT470_BG), if it is not set.
if validate.transferIsInvalid(clip):
clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT470_BG)
# setting color primaries info (to vs.PRIMARIES_BT470_BG), if it is not set.
if validate.primariesIsInvalid(clip):
clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT470_BG)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED)
# making sure frame rate is set to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# making sure the detected scan type is set (detected: top field first)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_TOP) # tff
# Deinterlacing using QTGMC
clip = havsfunc.QTGMC(Input=clip, Preset="Very Slow", TFF=True, opencl=True) # new fps: 50
# Making sure content is preceived as frame based
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive
clip = core.std.CropRel(clip=clip, left=12, right=12, top=2, bottom=10)# cropping to 696x564
# denoising using HQDN3D
clip = core.hqdn3d.Hqdn3d(clip=clip, lum_spac=2.00, chrom_spac=2.00, lum_tmp=4.00, chrom_tmp=6.00)
# Resizing using 10 - bicubic spline
clip = core.fmtc.resample(clip=clip, kernel="spline16", w=720, h=548, interlaced=False, interlacedd=False) # resolution 720x548 before YUV420P8 after YUV420P16
# adjusting output color from: YUV420P16 to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited", dither_type="error_diffusion")
# set output frame rate to 50fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=50, fpsden=1)
# output
clip.set_output()
encoding worked fine here. (horribly slow due to QTGMC(Input=clip, Preset="Very Slow"), but it worked fine. No clue why you would use 'Very Slow' on such a source)
(adding the experimental cut support with your values, didn't change a thing, encoding worked fine too)
Out of curiosity (I doubt this has anything to do with the crash):
What settings do you enable aside from:
are you using for Hybrid to think the output should be 720x576?
For me the:
# fallback - adjusting from 696x576 to output resolution 720x576
clip = core.resize.Spline64(clip=clip, width=720, height=576) # resolution 720x576)
part in your script does not appear.
For some reason Hybrid on your end does seem to think the output should be 720x576, but I don't see why.
Cu Selur
Ps.: side note about the source filtering: ChromaShiftSP with x=2 and SpotLess with temporal radius 3 and some deblocking and dehaloing should be applied.
Ps.: Really try to disable / uninstall that antivirus software.
----
Dev versions are in the 'experimental'-folder of my GoogleDrive, which is linked on the download page.
