[BUG] SVT-AV1 encoding end immediately

[Werve]

Hello, I noticed another problem with AV1 encoding, this time with SVT-AV1

• What Hybrid version did you use? Windows 10 21H2 Hybrid dev 2022.04.09.1
  
• What did happen? If an error message came up what did it exactly say? In the "Jobs" tab it is written Progress 50% State Finished (without output file created)
  
• What did you do to get to the bug? Just load the video input, set to use SVT-AV1 and load a saved configuration (but same problem with default one) then start encoding
  
• Debug output attached
  

[Selur]

Will look at it in a few hours after work.

[Werve]

Thanks. 
Take your time, there is no hurry.
Just the fact that you are always available to fix bugs and listen to requests on the forum means a lot, hard to find elsewhere.

[Selur]

Okay, totally missed that the SVT developers changed a bunch of parameters.
-> this will take a while and SVT-HEVC is probably broken too.

[Selur]

Okay, since tons of the parameters changed I will probably have to rewrite the whole thing or I drop support for it.

Options:
      --help                       Shows the command line options currently available
      --version                    Shows the version of the library that's linked to the library
  -i, --input                      Input raw video (y4m and yuv) file path, use `stdin` to read from pipe
  -b, --output                     Output compressed (ivf) file path, use `stdout` to write to pipe
  -c, --config                     Configuration file path  
      --errlog                     Error file path, defaults to stderr
  -o, --recon                      Reconstructed yuv file path
      --stat-file                  PSNR / SSIM per picture stat output file path, requires `--enable-stat-report 1`
      --pred-struct-file           Manual prediction structure file path
      --progress                   Verbosity of the output, default is 1 [0: no progress is printed, 2: aomenc style output]
      --no-progress                Do not print out progress, default is 0 [1: `--progress 0`, 0: `--progress 1`]
      --preset                     Encoder preset, presets < 0 are for debugging. Higher presets means faster encodes, but with a quality tradeoff, default is 10 [-2-13]
      --svtav1-params              colon separated list of key=value pairs of parameters with keys based on config file options

Encoder Global Options:
  -w, --width                      Frame width in pixels, inferred if y4m, default is 0 [64-16384]
  -h, --height                     Frame height in pixels, inferred if y4m, default is 0 [64-8704]
      --forced-max-frame-width     Maximum frame width value to force, default is 0 [64-16384]
      --forced-max-frame-height    Maximum frame height value to force, default is 0 [64-8704]
  -n, --frames                     Number of frames to encode. If `n` is larger than the input, the encoder will loop back and continue encoding, default is 0 [0: until EOF, 1-`(2^63)-1`]
      --nb                         Buffer `n` input frames into memory and use them to encode, default is -1 [-1: no frames buffered, 1-`(2^31)-1`]
      --color-format               Color format, only yuv420 is supported at this time, default is 1 [0: yuv400, 1: yuv420, 2: yuv422, 3: yuv444]
      --profile                    Bitstream profile, default is 0 [0: main, 1: high, 2: professional]
      --level                      Bitstream level, defined in A.3 of the av1 spec, default is 0 [0: autodetect from input, 2.0-7.3]
      --enable-hdr                 Enable writing of HDR metadata in the bitstream, default is 0 [0-1]
      --fps                        Input video frame rate, integer values only, inferred if y4m, default is 60 [1-240]
      --fps-num                    Input video frame rate numerator, default is 60000 [0-2^32-1]
      --fps-denom                  Input video frame rate denominator, default is 1000 [0-2^32-1]
      --input-depth                Input video file and output bitstream bit-depth, default is 8 [8, 10]
      --compressed-ten-bit-format    Pack 10bit video, handled between the app and library, default is 0 [0-1]
      --inj                        Inject pictures to the library at defined frame rate, default is 0 [0-1]
      --inj-frm-rt                 Set injector frame rate, only applicable with `--inj 1`, default is 60 [0-240]
      --enable-stat-report         Calculates and outputs PSNR SSIM metrics at the end of encoding, default is 0 [0-1]
      --asm                        Limit assembly instruction set, only applicable to x86, default is max [c, mmx, sse, sse2, sse3, ssse3, sse4_1, sse4_2, avx, avx2, avx512, max]
      --lp                         Target (best effort) number of logical cores to be used. 0 means all. Refer to Appendix A.1 of the user guide, default is 0 [0, core count of the machine]
      --pin                        Pin the execution to the first --lp cores. Overwritten to 0 when `--ss` is set. Refer to Appendix A.1 of the user guide, default is 1 [0-1]
      --ss                         Specifies which socket to run on, assumes a max of two sockets. Refer to Appendix A.1 of the user guide, default is -1 [-1, 0, -1]

Rate Control Options:
      --rc                         Rate control mode, default is 0 [0: CRF or CQP (if `--aq-mode` is 0), 1: VBR, 2: CBR]
  -q, --qp                         Initial QP level value, default is 35 [1-63]
      --crf                        Constant Rate Factor value, setting this value is equal to `--rc 0 --aq-mode 2 --qp x`, default is 35 [1-63]
      --tbr                        Target Bitrate (kbps), only applicable for VBR and CBR encoding, default is 7000 [1-100000]
      --mbr                        Maximum Bitrate (kbps) only applicable for CRF encoding, default is 0 [1-100000]
      --use-q-file                 Overwrite the encoder default picture based QP assignments and use QP values from `--qp-file`, default is 0 [0-1]
      --qpfile                     Path to a file containing per picture QP value separated by newlines
      --max-qp                     Maximum (highest) quantizer, only applicable for VBR and CBR, default is 63 [1-63]
      --min-qp                     Minimum (lowest) quantizer, only applicable for VBR and CBR, default is 1 [1-63]
      --aq-mode                    Set adaptive QP level, default is 2 [0: off, 1: variance base using AV1 segments, 2: deltaq pred efficiency]
      --use-fixed-qindex-offsets    Overwrite the encoder default hierarchical layer based QP assignment and use fixed Q index offsets, default is 0 [0-1]
      --key-frame-qindex-offset    Overwrite the encoder default keyframe Q index assignment, default is 0 [-256-255]
      --key-frame-chroma-qindex-offset    Overwrite the encoder default chroma keyframe Q index assignment, default is 0 [-256-255]
      --qindex-offsets             list of luma Q index offsets per hierarchical layer, separated by `,` with each offset in the range of [-256-255], default is `0,0,..,0`
      --chroma-qindex-offsets      list of chroma Q index offsets per hierarchical layer, separated by `,` with each offset in the range of [-256-255], default is `0,0,..,0`
      --luma-y-dc-qindex-offset    Luma Y DC Qindex Offset  
      --chroma-u-dc-qindex-offset    Chroma U DC Qindex Offset
      --chroma-u-ac-qindex-offset    Chroma U AC Qindex Offset
      --chroma-v-dc-qindex-offset    Chroma V DC Qindex Offset
      --chroma-v-ac-qindex-offset    Chroma V AC Qindex Offset
      --undershoot-pct             Allowable datarate undershoot (min) target (percentage), default is 25, but can change based on rate control [0-100]
      --overshoot-pct              Allowable datarate overshoot (max) target (percentage), default is 25, but can change based on rate control [0-100]
      --buf-sz                     Client buffer size (ms), only applicable for CBR, default is 6000 [0-10000]
      --buf-initial-sz             Client initial buffer size (ms), only applicable for CBR, default is 4000 [0-10000]
      --buf-optimal-sz             Client optimal buffer size (ms), only applicable for CBR, default is 5000 [0-10000]
      --recode-loop                Recode loop level, refer to "Recode loop level table" in the user guide for more info [0: off, 4: preset based]
      --bias-pct                   CBR/VBR bias, default is 50 [0: CBR-like, 1-99, 100: VBR-like]
      --minsection-pct             GOP min bitrate (expressed as a percentage of the target rate), default is 0 [0-100]
      --maxsection-pct             GOP max bitrate (expressed as a percentage of the target rate), default is 100 [0-100]

Multi-pass Options:
      --pass                       Multi-pass selection, pass 3 is only available for VBR, default is 0 [0: single pass encode, 1: first pass, 2: second pass, 3: third pass]
      --stats                      Filename for multi-pass encoding, default is "svtav1_2pass.log"
      --passes                     Number of encoding passes, default is preset dependent but generally 1 [1: one pass encode, 2: multi-pass encode]

GOP size and type Options:
      --keyint                     GOP size (frames), default is -2 [-2: ~5 seconds, -1: "infinite" and only applicable for CRF, 0: same as -1]
      --irefresh-type              Intra refresh type, default is 2 [1: FWD Frame (Open GOP), 2: KEY Frame (Closed GOP)]
      --scd                        Scene change detection control, default is 0 [0-1]
      --lookahead                  Number of frames in the future to look ahead, not including minigop, temporal filtering, and rate control, default is -1 [-1: auto, 0-120]
      --hierarchical-levels        Set hierarchical levels beyond the base layer, default is 4 [3: 4 temporal layers, 4: 5 layers, 5: 6 layers]
      --pred-struct                Set prediction structure, default is 2 [1: low delay frames, 2: random access]

AV1 Specific Options:
      --tile-rows                  Number of tile rows to use, `TileRow == log2(x)`, default changes per resolution but is 1 [0-6]
      --tile-columns               Number of tile columns to use, `TileCol == log2(x)`, default changes per resolution but is 1 [0-4]
      --enable-dlf                 Deblocking loop filter control, default is 1 [0-1]
      --enable-cdef                Enable Constrained Directional Enhancement Filter, default is 1 [0-1]
      --enable-restoration         Enable loop restoration filter, default is 1 [0-1]
      --enable-tpl-la              Temporal Dependency model control, currently forced on library side, only applicable for CRF/CQP, default is 1 [0-1]
      --enable-mfmv                Motion Field Motion Vector control, default is -1 [-1: auto, 0-1]
      --fast-decode                Fast Decoder levels, default is 0 [0-1]
      --enable-tf                  Enable ALT-REF (temporally filtered) frames, default is 1 [0-1]
      --enable-overlays            Enable the insertion of overlayer pictures which will be used as an additional reference frame for the base layer picture, default is 0 [0-1]
      --tune                       Specifies whether to use PSNR or VQ as the tuning metric [0 = VQ, 1 = PSNR], default is 1 [0-1]
      --scm                        Set screen content detection level, default is 2 [0: off, 1: on, 2: content adaptive]
      --rmv                        Restrict motion vectors from reaching outside the picture boundary, default is 0 [0-1]
      --film-grain                 Enable film grain, default is 0 [0: off, 1-50: level of denoising for film grain]
      --enable-dnl-denoising       Apply denoising when film grain is ON, default is 1 [0: no denoising, film grain data is still in frame header, 1: level of denoising is set by the film-grain parameter]
      --superres-mode              Enable super-resolution mode, refer to the super-resolution section in the user guide, default is 0 [0: off, 1-3, 4: auto-select mode]
      --superres-denom             Super-resolution denominator, only applicable for mode == 1, default is 8 [8: no scaling, 9-15, 16: half-scaling]
      --superres-kf-denom          Super-resolution denominator for key frames, only applicable for mode == 1, default is 8 [8: no scaling, 9-15, 16: half-scaling]
      --superres-qthres            Super-resolution q-threshold, only applicable for mode == 3, default is 43 [0-63]
      --superres-kf-qthres         Super-resolution q-threshold for key frames, only applicable for mode == 3, default is 43 [0-63]
      --sframe-dist                S-Frame interval (frames) (0: OFF[default], > 0: ON)
      --sframe-mode                S-Frame insertion mode ([1-2], 1: the considered frame will be made into an S-Frame only if it is an altref frame, 2: the next altref frame will be made into an S-Frame[default])

Color Description Options:
      --color-primaries            Color primaries, refer to Appendix A.2 of the user guide, default is 2 [0-12, 22]
      --transfer-characteristics    Transfer characteristics, refer to Appendix A.2 of the user guide, default is 2 [0-22]
      --matrix-coefficients        Matrix coefficients, refer to Appendix A.2 of the user guide, default is 2 [0-14]
      --color-range                Color range, default is 0 [0: Studio, 1: Full]
      --chroma-sample-position     Chroma sample position, default is 'unknown' ['unknown', 'vertical'/'left', 'colocated'/'topleft']
      --mastering-display          Mastering display metadata in the format of "G(x,y)B(x,y)R(x,y)WP(x,y)L(max,min)", refer to the user guide Appendix A.2
      --content-light              Set content light level in the format of "max_cll,max_fall", refer to the user guide Appendix A.2

Cu Selur

Ps.: yup that will take me a few days just to readup on stuff and a few weeks to implement so svt-av1 will be removed in the next release.
https://github.com/AOMediaCodec/SVT-AV1/...r_guide.md is quite a bit to read and I also will have to side check against https://aomediacodec.github.io/av1-spec/ to not add stuff that isn't fully specified. argh.

[Werve]

Okay, I understand.

Also x264 encoding crash "Crashed with exit status 0" (tried with 2 saved configuration)

[Selur]

x264 aborts with:

x264 [error]: invalid width x height (32768x18432)

encoding call itself:

"C:\Program Files\Hybrid\64bit\ffmpeg.exe" -y -loglevel fatal -noautorotate -nostdin -threads 8 -ignore_editlist true -i "D:\Videos\Download di RealPlayer\AR.Drone duel in Central Park - AR.FlyingAce FREE APP!(1080p_H.264-AAC).mp4" -map 0:0 -an -sn -vf scale=32768:18432,zscale=rangein=tv:range=tv -pix_fmt yuv420p -vsync 0 -sws_flags bicublin -f rawvideo - | "C:\Program Files\Hybrid\64bit\x264.exe" --preset slower --crf 20.00 --ref 9 --keyint 10000 --sync-lookahead 12 --qcomp 0.40 --qpmax 51 --partitions i4x4,i8x8,p8x8,b8x8 --merange 32 --subme 10 --psy-rd 1.00:0.15 --aq-mode 3 --sar 1:1 --deblock -1:-1 --non-deterministic --range tv --colormatrix bt470bg --demuxer raw --input-res 32768x18432 --input-csp i420 --input-range tv --input-depth 8 --fps 25/1 --opencl --opencl-device 1 --opencl-clbin "C:\Users\Antonio\AppData\Local\Temp\x264_lookahead_2022-04-14@21_16_51_2510_02.clbin" --output-depth 8 --output "C:\Users\Antonio\AppData\Local\Temp\2022-04-14@21_16_51_2510_02.264" -

looks correct.

My guess is that x264 does not support such a high resolution.
How much RAM do you have? 16k will take over 32gb,...
"--keyint 10000" <- why not use open-gop?
Also 'opencl-clbin' also will not work with such a resolution, since the vram requirements should also be humongous.

I suspect that either x264 simply doesn't support the resolution (couldn't find anything in the x264 source code so far) or you are simply running out of RAM.
Even if I disable cabac and use the far less (effective and ressource friendly cavlc by using base profile and way faster settings) I get 99% RAM usge on a 32gb machine. (it does encode)
So seeing that you are using monsterous settings your system probably would need more than 128GB if you used base profile with simple settings, your settings probably need more double that in RAM.
-> I don't think this is bug in Hybrid.

Cu Selur

Ps.: Also note that afaik current hardware decoders are limited to max 8192x8192 so you might also in problems decoding 32k streams.

[Werve]

x264 aborts with:

x264 [error]: invalid width x height (32768x18432)

What?! Actually is the same video file I'm using for codec tests, a normal 1080p video.
I see In the resize tab somehow it changed to upscale to that resolution. Maybe the Maximize resolution option in Misc sub-tab but that was checked in previous tests too.
In meanwhile I just did some test with SuperResolution upscaling (Vapoursynth tab).

"--keyint 10000" <- why not use open-gop?

Open-gop can be over 10000? Didn't want push too much, but anyway it is and old configuration and I think is not worth the quality change use so high value/open-gop so I'll just remove.

I suspect that either x264 simply doesn't support the resolution (couldn't find anything in the x264 source code so far) or you are simply running out of RAM.
Even if I disable cabac and use the far less (effective and ressource friendly cavlc by using base profile and way faster settings) I get 99% RAM usge on a 32gb machine. (it does encode)
So seeing that you are using monsterous settings your system probably would need more than 128GB if you used base profile with simple settings, your settings probably need more double that in RAM.

Yes, me too have 32 GB RAM (but a lot already in use) but the encoding immediately stopped so the encoder has probably given up due to the available RAM.

So sorry probably is the right behavior, just didn't' realize it changed in the resize tab.

[Selur]

Open-gop can be over 10000? Didn't want push too much, but anyway it is and old configuration and I think is not worth the quality change use so high value/open-gop so I'll just remove.

OpenGOP only uses IDR frame if it makes sense for the compressibility, so you can end up with whole clips only have one IDR frame at the beginning.

Maybe the Maximize resolution option in Misc sub-tab but that was checked in previous tests too.

Yes, thats the culprit, that option only makes sense if you use a profile that is restricted otherwise Hybrid will resize to the max resolution it does support and that is 32768x32768.

Cu Selur

[Selur]

Side note: most of the stuff mentioned there doesn't work.
-> will probably have working SVT-AV1 encoding back tomorrow.

Cu Selur