Random Crashes when processing videos.

[QLaHPD]

I've been using Hybrid for quite a while now, and for as long as I can remember, there have been occasional crashes when processing videos in a job queue, especially during 2-pass encoding jobs, regardless of the codec used.

For example, I'm currently processing a batch of 370 videos using aomenc, and some of them crash during the encoding process. Below, I’ve included the full command used and details about one of the videos that failed.



aomenc --passes=2 --pass=1 --target-bitrate=48 --end-usage=vbr --fpf="/media/ramdisk/hybrid_tmp/zBnFJjKqVM8_20200124.stats" --profile=0 --cpu-used=8 --min-q=1 --max-q=63 --lag-in-frames=25 --bias-pct=70 --minsection-pct=15 --maxsection-pct=10000 --drop-frame=0 --undershoot-pct=0 --overshoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --arnr-maxframes=7 --arnr-strength=5 --sharpness=0 --static-thresh=0 --noise-sensitivity=0 --tune-content=default --tile-columns=0 --tile-rows=0 --mtu-size=0 --frame-boost=0 --aq-mode=1 --min-gf-interval=0 --max-gf-interval=0 --gf-min-pyr-height=0 --gf-max-pyr-height=5 --threads=32 --width=128 --height=72 --input-bit-depth=10 --bit-depth=10 --denoise-noise-level=0 --min-partition-size=4 --max-partition-size=128 --fp-mt=1 --sb-size=dynamic --max-reference-frames=7 --disable-trellis-quant=0 --enable-tpl-model=1 --enable-keyframe-filtering=0 --force-video-mode=1 --reduced-tx-type-set=1 --use-intra-dct-only=0 --use-inter-dct-only=0 --use-intra-default-tx-only=0 --quant-b-adapt=1 --coeff-cost-upd-freq=0 --mode-cost-upd-freq=0 --mv-cost-upd-freq=0 -o NUL -

aomenc --passes=2 --pass=2 --target-bitrate=48 --end-usage=vbr --fpf="/media/ramdisk/hybrid_tmp/zBnFJjKqVM8_20200124.stats" --profile=0 --cpu-used=8 --min-q=1 --max-q=63 --lag-in-frames=25 --bias-pct=70 --minsection-pct=15 --maxsection-pct=10000 --drop-frame=0 --undershoot-pct=0 --overshoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --arnr-maxframes=7 --arnr-strength=5 --sharpness=0 --static-thresh=0 --noise-sensitivity=0 --tune-content=default --tile-columns=0 --tile-rows=0 --mtu-size=0 --frame-boost=0 --aq-mode=1 --min-gf-interval=0 --max-gf-interval=0 --gf-min-pyr-height=0 --gf-max-pyr-height=5 --threads=32 --width=128 --height=72 --input-bit-depth=10 --bit-depth=10 --denoise-noise-level=0 --min-partition-size=4 --max-partition-size=128 --fp-mt=1 --sb-size=dynamic --max-reference-frames=7 --disable-trellis-quant=0 --enable-tpl-model=1 --enable-keyframe-filtering=0 --force-video-mode=1 --reduced-tx-type-set=1 --use-intra-dct-only=0 --use-inter-dct-only=0 --use-intra-default-tx-only=0 --quant-b-adapt=1 --coeff-cost-upd-freq=0 --mode-cost-upd-freq=0 --mv-cost-upd-freq=0 -o "/media/ramdisk/hybrid_tmp/zBnFJjKqVM8_20200124.ivf" -



The video in question was downloaded directly from YouTube using yt-dlp without doing any extra modifications, via this link: 
https://www.youtube.com/watch?v=zBnFJjKqVM8

As I mentioned, this issue occurs not only with AV1 encoding using aomenc, but also with other codecs like x265, other AV1 encoders, and even VPX. When I switch to single-pass quantized mode, the encoding usually completes without crashing. However, I really need fine control over the bitrate, so 2-pass encoding is important for me.

I'm using the last main version of Hybrid, my computer specs are the following:
i5-12500H
16GB RAM
RTX 3050 mobile 4GB

[Selur]

Sadly, I can't fix random crashes. I can only fix stuff that is reproducible.
Could be ram issues (especially when using a ram disk) other tools interfering, bug in the tool versions you use,...
Does the crash always happen if you process the file from that link? You should at least upload the file somewhere, where I can download the file normally to be sure the problem isn't just from some downloading glitch that caused a broken file.
Especially with a debug output, I have no clue about the decoding call and other settings.
Read the sticky on what info is needed to reproduce/fix bugs.
=> with the given information, there is nothing more I can do

Cu Selur

[QLaHPD]

Of course, to clarify, when I said "random errors," I meant that, to me, there doesn't seem to be any specific pattern in the input videos, randomly, some videos (and it's always the same ones) crash during the execution of pass 2 of aomenc.

I don't believe it's anything related to RAM, many GBs are free, and the RAM disk is seen as a normal drive by the programs. I use it so I don't wear out my SSD with temporary files, since I process a lot of videos.

I'll leave an example video on SendGB where you can download and test it:
https://sendgb.com/rAJFXsYpdVi

I'm using the latest version of Hybrid, and the commands that caused the crash are the ones I shared in the original message.

[Selur]

You only shared your encoding call parameters,...

Using:

aomenc --passes=2 --pass=1 --target-bitrate=48 --end-usage=vbr --fpf="J:\tmp\zBnFJjKqVM8_20200124.stats" --profile=0 --cpu-used=8 --min-q=1 --max-q=63 --lag-in-frames=25 --bias-pct=70 --minsection-pct=15 --maxsection-pct=10000 --drop-frame=0 --undershoot-pct=0 --overshoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --arnr-maxframes=7 --arnr-strength=5 --sharpness=0 --static-thresh=0 --noise-sensitivity=0 --tune-content=default --tile-columns=0 --tile-rows=0 --mtu-size=0 --frame-boost=0 --aq-mode=1 --min-gf-interval=0 --max-gf-interval=0 --gf-min-pyr-height=0 --gf-max-pyr-height=5 --threads=64 --width=128 --height=72 --input-bit-depth=10 --bit-depth=10 --denoise-noise-level=0 --min-partition-size=4 --max-partition-size=128 --fp-mt=1 --sb-size=dynamic --max-reference-frames=7 --disable-trellis-quant=0 --enable-tpl-model=1 --enable-keyframe-filtering=0 --force-video-mode=1 --reduced-tx-type-set=1 --use-intra-dct-only=0 --use-inter-dct-only=0 --use-intra-default-tx-only=0 --quant-b-adapt=1 --coeff-cost-upd-freq=0 --mode-cost-upd-freq=0 --mv-cost-upd-freq=0 -o NUL -

aomenc --passes=2 --pass=2 --target-bitrate=48 --end-usage=vbr --fpf="J:\tmp\zBnFJjKqVM8_20200124.stats" --profile=0 --cpu-used=8 --min-q=1 --max-q=63 --lag-in-frames=25 --bias-pct=70 --minsection-pct=15 --maxsection-pct=10000 --drop-frame=0 --undershoot-pct=0 --overshoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --arnr-maxframes=7 --arnr-strength=5 --sharpness=0 --static-thresh=0 --noise-sensitivity=0 --tune-content=default --tile-columns=0 --tile-rows=0 --mtu-size=0 --frame-boost=0 --aq-mode=1 --min-gf-interval=0 --max-gf-interval=0 --gf-min-pyr-height=0 --gf-max-pyr-height=5 --threads=64 --width=128 --height=72 --input-bit-depth=10 --bit-depth=10 --denoise-noise-level=0 --min-partition-size=4 --max-partition-size=128 --fp-mt=1 --sb-size=dynamic --max-reference-frames=7 --disable-trellis-quant=0 --enable-tpl-model=1 --enable-keyframe-filtering=0 --force-video-mode=1 --reduced-tx-type-set=1 --use-intra-dct-only=0 --use-inter-dct-only=0 --use-intra-default-tx-only=0 --quant-b-adapt=1 --coeff-cost-upd-freq=0 --mode-cost-upd-freq=0 --mv-cost-upd-freq=0 -o "J:\tmp\zBnFJjKqVM8_20200124.ivf" -

which, unless I missed something, should line up with your settings, encoding worked fine for me.

=> I need a debug output of when it crashes, so I can see what decoder calls, etc. are used and maybe even see an error in the processing.

Cu Selur

[QLaHPD]

How do I generate this debug file?

---

Never mind, I found it

[Selur]

According to the debug output, the first pass

"/home/visilionosh/to_storage/TOOLS/hybrid/ffmpeg" -y -loglevel fatal -noautorotate -nostdin -threads 8 -i "/mnt/wwn-0x50014ee267906e25/UCJm2TgUqtK1_NLBrjNQ1P-w/zBnFJjKqVM8_20200124.webm" -map 0:0 -an -sn -vf scale=256:144 -pix_fmt yuv420p10le -strict -1 -fps_mode passthrough -sws_flags spline -f yuv4mpegpipe - | "/home/visilionosh/to_storage/TOOLS/hybrid/aomenc" --passes=2 --pass=1 --target-bitrate=48 --end-usage=vbr --fpf=/media/ramdisk/hybrid_tmp/zBnFJjKqVM8_20200124_1_2025-03-22@12_31_09_8710_04.stats --profile=0 --cpu-used=8 --min-q=1 --max-q=63 --lag-in-frames=25 --bias-pct=70 --minsection-pct=15 --maxsection-pct=10000 --drop-frame=0 --undershoot-pct=0 --overshoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --arnr-maxframes=7 --arnr-strength=5 --sharpness=0 --static-thresh=0 --noise-sensitivity=0 --tune-content=default --tile-columns=0 --tile-rows=0 --mtu-size=0 --frame-boost=0 --aq-mode=1 --min-gf-interval=0 --max-gf-interval=0 --gf-min-pyr-height=0 --gf-max-pyr-height=5 --threads=32 --width=256 --height=144 --input-bit-depth=10 --bit-depth=10 --denoise-noise-level=0 --min-partition-size=4 --max-partition-size=128 --fp-mt=1 --sb-size=dynamic --max-reference-frames=7 --disable-trellis-quant=0 --enable-tpl-model=1 --enable-keyframe-filtering=0 --force-video-mode=1 --reduced-tx-type-set=1 --use-intra-dct-only=0 --use-inter-dct-only=0 --use-intra-default-tx-only=0 --quant-b-adapt=1 --coeff-cost-upd-freq=0 --mode-cost-upd-freq=0 --mv-cost-upd-freq=0 -o NUL -

finished after 00:00:00.467. This seems too fast, but since it didn't throw an error and the .stats file was created, Hybrid does assume it is correct.

The second pass:

"/home/visilionosh/to_storage/TOOLS/hybrid/ffmpeg" -y -loglevel fatal -noautorotate -nostdin -threads 8 -i "/mnt/wwn-0x50014ee267906e25/UCJm2TgUqtK1_NLBrjNQ1P-w/zBnFJjKqVM8_20200124.webm" -map 0:0 -an -sn -vf scale=256:144 -pix_fmt yuv420p10le -strict -1 -fps_mode passthrough -sws_flags spline -f yuv4mpegpipe - | "/home/visilionosh/to_storage/TOOLS/hybrid/aomenc" --passes=2 --pass=2 --target-bitrate=48 --end-usage=vbr --fpf=/media/ramdisk/hybrid_tmp/zBnFJjKqVM8_20200124_1_2025-03-22@12_31_09_8710_04.stats --profile=0 --cpu-used=8 --min-q=1 --max-q=63 --lag-in-frames=25 --bias-pct=70 --minsection-pct=15 --maxsection-pct=10000 --drop-frame=0 --undershoot-pct=0 --overshoot-pct=0 --buf-sz=6 --buf-initial-sz=4 --buf-optimal-sz=5 --drop-frame=0 --kf-min-dist=0 --kf-max-dist=250 --auto-alt-ref=1 --arnr-maxframes=7 --arnr-strength=5 --sharpness=0 --static-thresh=0 --noise-sensitivity=0 --tune-content=default --tile-columns=0 --tile-rows=0 --mtu-size=0 --frame-boost=0 --aq-mode=1 --min-gf-interval=0 --max-gf-interval=0 --gf-min-pyr-height=0 --gf-max-pyr-height=5 --threads=32 --width=256 --height=144 --input-bit-depth=10 --bit-depth=10 --denoise-noise-level=0 --min-partition-size=4 --max-partition-size=128 --fp-mt=1 --sb-size=dynamic --max-reference-frames=7 --disable-trellis-quant=0 --enable-tpl-model=1 --enable-keyframe-filtering=0 --force-video-mode=1 --reduced-tx-type-set=1 --use-intra-dct-only=0 --use-inter-dct-only=0 --use-intra-default-tx-only=0 --quant-b-adapt=1 --coeff-cost-upd-freq=0 --mode-cost-upd-freq=0 --mv-cost-upd-freq=0 -o "/media/ramdisk/hybrid_tmp/2025-03-22@12_31_09_8710_05.ivf" -

crashes with:

aomenc output: Failed to initialize encoder: Invalid parameter
    rc_twopass_stats_in requires at least two packets.

Which usually indicates that the stats file is broken.
If you call the 1st pass call inside a terminal, do you see any errors?
Does '/media/ramdisk/hybrid_tmp/zBnFJjKqVM8_20200124_1_2025-03-22@12_31_09_8710_04.stats' contain any data or is it 0 bytes in size? (Hybrid does only check whether the file exists)
Since audio encoding is run in parallel, does it also crash for you if 'Parallel subjob processing' is disabled?
Does it work if the temp folder isn't in a ram disk?

Cu Selur

[QLaHPD]

Did you mean calling the command in a terminal using ffmpeg`? I tried doing that, but I’m not sure if running it with ffmpeg is the same as what Hybrid does.

I haven’t tried using aomenc directly:

"
(base) visilionosh@visilionosh:~$ aomenc
Command 'aomenc' not found, but can be installed with:
sudo apt install aom-tools
"
Should I try?


Yes, the .stats file is 0 bytes.

Yes, the error also occurs when run sequentially.

No, it still crashes even when not using a RAM disk.


Edit: don't know if helps, but chatgpt said:
The log shows that when Hybrid builds the aomenc command it relies on metadata from the input file—such as codec information and keyframe positions—to properly process and segment the video stream. However, the log repeatedly notes that there’s “no Codec” and “no keyFramePositions” for the input webm file (​).

[Selur]

You should try the command I posted and not install another one.
Your Hybrid version did come with an aomenc binary.
Assuming you figured that out yourself and the responses are based on the command line I posted.
Does calling:

"/home/visilionosh/to_storage/TOOLS/hybrid/ffmpeg" -y -noautorotate -nostdin -threads 8 -i "/mnt/wwn-0x50014ee267906e25/UCJm2TgUqtK1_NLBrjNQ1P-w/zBnFJjKqVM8_20200124.webm" -map 0:0 -an -sn -vf scale=256:144 -pix_fmt yuv420p10le -strict -1 -fps_mode passthrough -sws_flags spline -f yuv4mpegpipe /dev/null

Process the whole file, or does it report any errors or warnings?

The log shows that when Hybrid builds the aomenc command it relies on metadata from the input file—such as codec information and keyframe positions—to properly process and segment the video stream. However, the log repeatedly notes that there’s “no Codec” and “no keyFramePositions” for the input webm file (​).

No, chatgpt has no clue what it's writing about. (the input is fed to aomenc through a pipe, there are no keyframe positions&co,..)

Cu Selur

[QLaHPD]

Okay, now I get it. Yes, when I execute this command by terminal I also get the same error, and when I execute this last one (/dev/null) it returns an interesting log, I'll attach it in full, but basically many lines repeating the same error and some with this:

[vist#0:0/av1 @ 0x6319fdb4c980] Error submitting packet to decoder: Function not implemented                                                                                                                                               
[av1 @ 0x6319fdb83540] Your platform doesn't support hardware accelerated AV1 decoding.                                                                                                                                                     

Is the problem caused by a lack of support for some instruction?

[Selur]

[av1 @ 0x6319fdb83540] Your platform doesn't support hardware accelerated AV1 decoding.

Doesn't seem like a problem, since the decoder call never requested hardware decoding, so ffmpeg should use software decoding.

[av1 @ 0x6319fdb83540] Failed to get pixel format.
[vist#0:0/av1 @ 0x6319fdb4c980] Error submitting packet to decoder: Function not implemented
[av1 @ 0x6319fdb83540] Missing Sequence Header.
[vist#0:0/av1 @ 0x6319fdb4c980] Error submitting packet to decoder: Invalid data found when processing input

seems more like an issue

If you also have a proper setup Vapoursynth environment (suitable to Hybrid), you can try whether it helps to enable 'Filtering->Vapourynth->Misc->Script->Always use Vapoursynth'.
Other than that, you could try a newer ffmpeg build.(newer releases of Hybrid do come with newer builds, but you could also install another ffmpeg version on your system and point Hybrid to it)

Cu Selur