SVT-AV1 Unexpected Behaviour

[BCJohn]

Hi Selur,

I'm running Hybrid 2023.01.22.1 on Windows 11 64bit 22H2 and having some strange behavior when encoding HDR10 video using the SVT-AV1 encoder.
I'm getting drastically larger (3.5x input) output file sizes than expected.

With a 1min30 cut of a 2160p DV(HDR10 fallback) file as input and CRF18/Preset 7 as the main settings along with the below the file size goes from an input of 910mb to an output of 3.1gb.

SvtAv1EncApp --input stdin  --enable-hdr 1 --width 3840 --height 2160 --fps-num 24000 --fps-denom 1001 --input-depth 10 --profile 0 --tile-rows 0 --tile-columns 0 --rc 0 --crf 18 --mbr 0 --aq-mode 2 --preset 7 --irefresh-type 1 --lookahead 33 --scm 0 --use-q-file 1 --qpfile GENERATED_QP_FILE --color-primaries 9 --transfer-characteristics 16 --matrix-coefficients 9 --color-range 0 --mastering-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50)" --content-light 1000,227 --output "E:\###DoVi_Scripts\###x265 Test Configs\HDR10 DV Source_new.av1"

Doing a comparison to Handbrake using similar settings of CRF18 and Preset 7 results in that same 910mb having an output of 72mb.

Looking at the debug log for Hybrid it also looks like the Master Display settings aren't passed through properly when grabbing from input (likely different format than input) and the setting for Tune of VQ is ignored (always results in PSNR as the tune).

Svt[info]: SVT [version]:    SVT-AV1 Encoder Lib v1.4.1-35-gad82cde15
Svt[info]: SVT [build]  :    GCC 12.2.0     64 bit
Svt[info]: LIB Build date: Jan 19 2023 19:10:17
Svt[info]: -------------------------------------------
Svt[warn]: Invalid mastering display info will be clipped to 0.0 to 1.0
Svt[warn]: Fwd key frame is only supported for hierarchical levels 4 at this point. Hierarchical levels are set to 4
Svt[info]: Number of logical cores available: 32
Svt[info]: Number of PPCS 89
Svt[info]: [asm level on system : up to avx2]
Svt[info]: [asm level selected : up to avx2]
Svt[info]: -------------------------------------------
Svt[info]: SVT [config]: main profile    tier (auto)    level (auto)
Svt[info]: SVT [config]: width / height / fps numerator / fps denominator         : 3840 / 2160 / 24000 / 1001
Svt[info]: SVT [config]: bit-depth / color format                     : 10 / YUV420
Svt[info]: SVT [config]: preset / tune / pred struct                     : 7 / PSNR / random access
Svt[info]: SVT [config]: gop size / mini-gop size / key-frame type             : 160 / 16 / FWD key frame
Svt[info]: SVT [config]: BRC mode / rate factor                     : CRF / 18
Svt[info]: -------------------------------------------

I tested with a regular HDR10 clip with no DV metadata and had a similar result, with output size exceeding input by a comparable margin.

In the activity output in Handbrake it looks like both Hybrid and Handbrake have the same CRF & Preset but with slightly different versions/builds of SVT and a few negligible (when considering file size outputs) differences in settings.

Svt[info]: SVT [version]:    SVT-AV1 Encoder Lib v1.4.1
Svt[info]: SVT [build]  :    GCC 10.2.0     64 bit
Svt[info]: LIB Build date: Jan 23 2023 17:12:39
Svt[info]: -------------------------------------------
Svt[info]: Number of logical cores available: 32
Svt[info]: Number of PPCS 88
Svt[info]: [asm level on system : up to avx2]
Svt[info]: [asm level selected : up to avx2]
Svt[info]: -------------------------------------------
Svt[info]: SVT [config]: main profile    tier (auto)    level (auto)
Svt[info]: SVT [config]: width / height / fps numerator / fps denominator         : 3840 / 2160 / 24000 / 1001
Svt[info]: SVT [config]: bit-depth / color format                     : 10 / YUV420
Svt[info]: SVT [config]: preset / tune / pred struct                     : 7 / VQ / random access
Svt[info]: SVT [config]: gop size / mini-gop size / key-frame type             : 244 / 16 / key frame
Svt[info]: SVT [config]: BRC mode / rate factor                     : CRF / 18
Svt[info]: -------------------------------------------

I've also tried with a 1080p SDR (BT.709) mkv file as input and the results were completely as expected in terms of compression going from 106mb to 40mb(+/- 2mb) for both Hybrid and Handbrake, so I'm wondering if this is an issue with HDR10 (and/or associated settings) specifically.

I've attached the Debug Output from the 1min30 HDR10 DV clip and can provide the input video file if you want it.

Help would be greatly appreciated.
Thanks!

[Selur]

If you want smaller file size:
a. use a higher crf
b. deblock, maybe denoise content.

Cu Selur

[BCJohn]

2160p HDR10 Input video 49.3 Mb/s
CRF50 Preset 7 in Hybrid using SVT-AV1
Output video 446 Mb/s

CRF50 Preset 7 in Handbrake using SVT-AV1
Output video 515 kb/s


1080p SDR Input video 18.4 Mb/s
CRF50 Preset 7 in Hybrid using SVT-AV1
Output video 85.5 Mb/s

CRF50 Preset 7 in Handbrake using SVT-AV1
Output video 583 kb/s

All filters (denoise, deblock etc) in both Handbrake and Hybrid were off.
Quality difference is obviously significantly better with Hybrid output, but does not match the poor quality you'd expect from a ridiculously high CRF.

There does not seem to be a CRF value to use for SVT-AV1 in Hybrid that will result in a file size smaller than input (with exception to animated content, which has comparable output bitrate to Handbrake)

[Selur]

I tried reproducing the issue using:

"i:\Hybrid\64bit\ffmpeg.exe" -y -loglevel fatal -noautorotate -nostdin -threads 8 -ignore_editlist true -i "G:\TestClips&Co\files\HDR\HDR10\4K sun HDR test.mp4" -map 0:0 -an -sn -vf zscale=rangein=tv:range=tv -pix_fmt yuv420p10le -strict -1 -vsync 0 -f yuv4mpegpipe - | "i:\Hybrid\64bit\SvtAv1EncApp.exe" --input stdin --enable-hdr 1 --width 3840 --height 2160 --fps-num 24000 --fps-denom 1001 --input-depth 10 --profile 0 --tile-rows 0 --tile-columns 0 --rc 0 --crf 18 --mbr 0 --aq-mode 2 --preset 7 --irefresh-type 1 --lookahead 33 --scm 0 --color-primaries 9 --transfer-characteristics 16 --matrix-coefficients 9 --chroma-sample-position 2 --color-range 0 --mastering-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50)" --content-light 1000,227 --output "g:\Output\2023-01-29@09_02_26_3710_12.av1"

which is basically what your call looks like (I just changed the paths and removed the qp file.) And here the output is way smaller than the input.

Can't reproduce this here at all.
I send you a link to my current dev version to be sure we use the same encoder versions etc.
In case that version does not fix the issue for you, I would need a sample of your source which allows reproducing the issue.

Cu Selur

[manzing]

Hi, I just tried a SVT-AV1 encoding and faced a similar issue using Hybrid dev 2023.02.09

Took a 1mn vidéo (236 MB), set a CQ 32 / speed 7/ AV1, and i got a 1.3 GB file 
I tested with another software (that rely on SvtAv1EncApp.exe too) using exact same settings and got a 13.6 MB file.

No HDR, 8 bit test is smaller but still huge (800 MB)

[Selur]

Trying to reproduce this I used a random clip, set 'Quantization strength' to 32, disabled 'CRF', set 'Bit depth' to 8-Bit and 'Speed preset' to 7.
SvtAV1EncApp call for me is:

SvtAv1EncApp --input stdin  --width 1920 --height 1080 --fps-num 25 --fps-denom 1 --profile 0 --tile-rows 0 --tile-columns 0 --rc 0 --qp 32 --aq-mode 0 --preset 7 --irefresh-type 1 --tune 0 --scm 0 --matrix-coefficients 1 --color-range 0 --output "J:\tmp\00340.av1"

(I tried different files and didn't have any issues here,...)
=> can't reproduce this here at all, for me output size is always way smaller than the input
Share proper details, preferably a debut output, if you need help.

Cu Selur

[manzing]

Interesting, mine is different, and I don't know where the --use-q-file and --qpfile come from.
Note that BCJohn mentionned it too, but obviously your call doesn't, maybe this is the answer ?

SvtAv1EncApp --input stdin  --width 1920 --height 1080 --fps-num 24 --fps-denom 1 --input-depth 10 --profile 0 --tile-rows 0 --tile-columns 0 --rc 0 --qp 32 --aq-mode 0 --preset 7 --irefresh-type 1 --tune 0 --scm 0 --use-q-file 1 --qpfile GENERATED_QP_FILE --matrix-coefficients 1 --color-range 0 --output "C:\Temp\test.av1"

[Selur]

don't know where the --use-q-file and --qpfile come from.

Old code, will be removed by Hybrid later on (old code), doesn't do anything.

--irefresh-type 1

Your input is HDR?

[manzing]

No, SDR input.

[Selur]

Ah, sorry my bad, mixed up parameters in my mind there.
=> No, clue what's causing the output to be so large for you. Issues does not happen here. Like I wrote before:

Share proper details, preferably a debut output, if you need help.

Cu Selur