Crash in AVC encoding pass 1

[incans]

Hardware: Core i7-4770K, 16GB RAM
OS: Windows 7 Pro 64bit
Hybrid: 2020.12.13.1

New Hybrid user still finding my way around.

This is my first attempt at trying to clean up some rough source material. Unfortunately it crashes early in the processing of the video.

Source file: makemkv rip of "James Taylor - Squibnocket (2006)" DVD
Source Issues: Flickering in the top ~6 lines of the frame, lots of grain (low light)

Processing with the default params I would use for a typical film DVD just reproduced the issues in the source, plus added a nasty judder in horizontal panning shots.

I added the following processing params:

X264
Bitrate: 2000
Tune: grain

Crop/Resize
Avisynth style: L=16, T=8, B=8, R=16

Filtering
Denoise: hqdn3D - slow, strong
Unsharpen: Strength=1.00, Radius=7x7
DeShake: (default settings when enabled)

Symptoms

Job crashes in pass 1 of the x264 transcode

Log only shows "exit code 0", but debug file mentions incorrect parameters to one of the filter stages.

Log file attached. [attachment=1236]

[Selur]

General comment: It's usually a bad idea to mix FFmpeg and Vapoursynth/Avisynth filtering.

Only visible error is:

x264 output: y4m [error]: bad sequence header magic
x264 [error]: could not open input file `-'

which usually means that something went wrong with the filtering/decoding.

Looking at the debug output:
Vapoursynth script used:

# Imports
import vapoursynth as vs
core = vs.get_core()
# Loading Plugins
core.std.LoadPlugin(path="C:/Program Files/Other/Hybrid/64bit/vsfilters/DeinterlaceFilter/TDeintMod/TDeintMod.dll")
core.std.LoadPlugin(path="C:/Program Files/Other/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# source: 'S:\video\Rips\Music\James Taylor - Squibnocket (2006)\title_t00.mkv'
# current color space: YUV420P8, bit depth: 8, resolution: 720x576, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: top field first
# Loading S:\video\Rips\Music\James Taylor - Squibnocket (2006)\title_t00.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="S:/video/Rips/Music/James Taylor - Squibnocket (2006)/title_t00.mkv", format="YUV420P8", cache=0, prefer_hw=0)
# making sure input color matrix is set as 470bg
clip = core.resize.Point(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# Deinterlacing using TDeintMod
clip = core.tdm.TDeintMod(clip=clip, order=0)
# cropping the video to 688x560
clip = core.std.CropRel(clip=clip, left=16, right=16, top=8, bottom=8)
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

seems fine to me.

Looking at the encoding call:

"C:\Program Files\Other\Hybrid\64bit\Vapoursynth\vspipe.exe" "C:\Users\colin\AppData\Local Temp\encodingTempSynthSkript_2020-12-30@00_14_03_0010.vpy" - --y4m |  "C:\Program Files\Other\Hybrid\64bit\ffmpeg.exe" -y -loglevel fatal -noautorotate -nostdin -threads 8 -f yuv4mpegpipe -i - -an -sn -vf hqdn3d=5:4:7:6,deshake=x=-1:y=-1:w=-1:rx=-1:ry:-1:egde=16:blocksize=16:contrast=mirror:search=8,unsharp=7:7:1.00:7:7:1.00,zscale=rangein=tv:range=tv -pix_fmt yuv420p -vsync 0  -f yuv4mpegpipe -
2 | "C:\Program Files\Other\Hybrid\64bit\x264.exe" --preset veryfast --pass 1 --bitrate 2000 --profile high --level 4.1 --sync-lookahead 1 --qcomp 0.8 --rc-lookahead 40 --ipratio 1.1 --pbratio 1.1 --qpmax 51 --no-dct-decimate --weightp 2 --aq-strength 0.5 --deadzone-inter 6 --deadzone-intra 6 --sar 16:15 --qpfile "C:\Users\colin\AppData\Local\Temp\title_t00_2020-12-30@00_14_03_0010_05.qp" --deblock -2:-2 --non-deterministic --range tv --stats "C:\Users\colin\AppData\Local\Temp\title_t00_2020-12-30@00_14_03_0010_06.stats" --demuxer y4m  --input-range tv --fps 25/1 --output-depth 8 --output NUL -

I don't see any mistake directly, but my guess is that it's the deshake filter.

-> will do some testing and report back

Cu Selur

[Selur]

Yup, it's the deshake filer, it aborts with:

[deshake @ 00000284cabb8fc0] Value -1.000000 for parameter 'rx' out of range [0 - 64]
[deshake @ 00000284cabb8fc0] [Eval @ 000000d94e3fdd80] Undefined constant or missing '(' in 'ry'
[deshake @ 00000284cabb8fc0] Unable to parse option value "ry"
[Parsed_deshake_0 @ 00000284c9487600] Option 'egde' not found
[AVFilterGraph @ 00000284c948ad40] Error initializing filter 'deshake' with args 'x=-1:y=-1:w=-1:rx=-1:ry:-1:egde=16:blocksize=16:contrast=mirror:search=8'
Error reinitializing filters!

It's complaining about rx not inside 0-64.
The documentation (https://www.ffmpeg.org/ffmpeg-all.html#deshake) says:

Attempt to fix small changes in horizontal and/or vertical shift. This filter helps remove camera shake from hand-holding a camera, bumping a tripod, moving on a vehicle, etc.

The filter accepts the following options:

x
y
w
h

Specify a rectangular area where to limit the search for motion vectors. If desired the search for motion vectors can be limited to a rectangular area of the frame defined by its top left corner, width and height. These parameters have the same meaning as the drawbox filter which can be used to visualise the position of the bounding box.

This is useful when simultaneous movement of subjects within the frame might be confused for camera motion by the motion vector search.

If any or all of x, y, w and h are set to -1 then the full frame is used. This allows later options to be set without specifying the bounding box for the motion vector search.

Default - search the whole frame.
rx
ry

Specify the maximum extent of movement in x and y directions in the range 0-64 pixels. Default 16.
edge

Specify how to generate pixels to fill blanks at the edge of the frame. Available values are:

‘blank, 0’

Fill zeroes at blank locations
‘original, 1’

Original image at blank locations
‘clamp, 2’

Extruded edge value at blank locations
‘mirror, 3’

Mirrored edge at blank locations

Default value is ‘mirror’.
blocksize

Specify the blocksize to use for motion search. Range 4-128 pixels, default 8.
contrast

Specify the contrast threshold for blocks. Only blocks with more than the specified contrast (difference between darkest and lightest pixels) will be considered. Range 1-255, default 125.
search

Specify the search strategy. Available values are:

‘exhaustive, 0’

Set exhaustive search
‘less, 1’

Set less exhaustive search.

Default value is ‘exhaustive’.
filename

If set then a detailed log of the motion search is written to the specified file.

-> Yup, it's a bug in Hybrid I wrongly restricted rx/ry. (correction: I wrongly mapped the parameters. + syntax changed)
-> will fix and send you a link to a dev version once I'm finished.

Cu Selur

[Selur]

Send you a pm with a link to a dev version where the deshake filter should work.

Cu Selur

[incans]

First, thank you for the fast and detailed response. Amazing support 


Re:

>> General comment: It's usually a bad idea to mix FFmpeg and Vapoursynth/Avisynth filtering.

I hadn't even realised that's what I was doing. I guess it takes a naive user to really stress a system ;-)

>> Yup, it's the deshake filer, it aborts with...

Well the good thing is that means I can at least try a conversion without the deShake filter to get me going, so that's good information, thank you.

>> Send you a pm with a link to a dev version where the deshake filter should work.

Again, superb support, and much appreciated thank you!

[Selur]

Please, check the version from the pm and let me know if everything it fixed.

Cu Selur

[incans]

Please, check the version from the pm and let me know if everything it fixed.

Cu Selur

The good news is the conversion now runs to completion without an error, so thamk you for the remarkable turnround on fixing the bug  

The processed video still looks pretty terrible, but that is my problem to try to work out the appropriate filter settings to try to make the best of the poor quality source.

One odd thing, so far when using Hybrid I have generally seen the audio processing and file swapping etc tends to be either single threaded or IO-bound, so the CPU is running at 25-50% utilisation in those phases, but the video encoding passes (with the default "0 threads" setting to use the default threading) are entirely CPU bound, so the CPU sits nailed at 100% for 1-2 hours for a full feature film.

Running the pipeline for this DVD rip, with more filters involved, the CPU only seemed to reach about 50% utilisation. Obviously with more filtering you'd expect a longer processing time, but I wondered whether there were also some changes I should be making to the threading configuration to get the best throughput?

I think I am going to be running a lot of tests on this one DVD to try to get a decent output. Cropping and rescaling back to 720x576 removed the flickering in the top lines, but either this or the deShake causes bad tearing and juddering movement in the output, plus new artifacts on the left and right hand sides of the frame (where the original video seems to be letterboxed with back borders to transfer a 5x4 aspect ratio source (presumably from DV tape) onto a 720x576 digital version.

There's a vast amount I don't know about processing poor quaity source material. Many of the options in Hybrid are less than obvuous in their effect. If there are any good guides to the options available that would be good to know.

However, on the original bug report, that appears to be fixed, so a positive outcome

[Selur]

One odd thing, so far when using Hybrid I have generally seen the audio processing and file swapping etc tends to be either single threaded or IO-bound, so the CPU is running at 25-50% utilisation in those phases, but the video encoding passes (with the default "0 threads" setting to use the default threading) are entirely CPU bound, so the CPU sits nailed at 100% for 1-2 hours for a full feature film.

Nothing odd about it. Audio compression generally doesn't really use multithreading.

Running the pipeline for this DVD rip, with more filters involved, the CPU only seemed to reach about 50% utilisation. Obviously with more filtering you'd expect a longer processing time, but I wondered whether there were also some changes I should be making to the threading configuration to get the best throughput?

Some filters are slow others are fast, I doubt that tweaking the threading settings will change much.

As genera hint:
Do not use the ffmpeg filters. Anything that can be done with those filters can be (better) done with Avisynth/Vapoursynth.

If there are any good guides to the options available that would be good to know.

Nope. Avisynth and Vapoursynth filters have their own Preview which normally helps checking out the effect of the filters.

Cu Selur