4K transcoding problem with NVENC (outdated false limitation in Hybrid)

[Selur]

What does:

NVEncC --check-features

output on your system?

I get for example:

Environment Info OS : Windows 10 x64 (17134) CPU: AMD Ryzen 7 1800X Eight-Core Processor (8C/16T) RAM: Used 5469 MB, Total 32696 MB GPU: GeForce GTX 1070 Ti (19 EU) @ 1683 MHz (399.07) List of available features. Codec: H.264/AVC Max Bframes 4 B Ref Mode yes RC Modes 63 Field Encoding yes MonoChrome no FMO no Quater-Pel MV yes B Direct Mode yes CABAC yes Adaptive Transform yes Max Temporal Layers 0 Hierarchial P Frames no Hierarchial B Frames no Max Level 51 Min Level 1 4:4:4 yes Max Width 4096 Max Height 4096 Dynamic Resolution Change yes Dynamic Bitrate Change yes Forced constant QP yes Dynamic RC Mode Change no Subframe Readback yes Constrained Encoding yes Intra Refresh yes Custom VBV Bufsize yes Dynamic Slice Mode yes Ref Pic Invalidiation yes PreProcess no Async Encoding yes Max MBs 65536 MAX MB per sec 983040 Lossless yes SAO no Me Only Mode yes Lookahead yes AQ (temporal) yes Weighted Prediction yes Max LTR Frames 8 10bit depth no Codec: H.265/HEVC Max Bframes 0 RC Modes 63 Field Encoding no MonoChrome no Quater-Pel MV yes B Direct Mode no Max Temporal Layers 0 Hierarchial P Frames no Hierarchial B Frames no Max Level 62 Min Level 1 4:4:4 yes Max Width 8192 Max Height 8192 Dynamic Resolution Change yes Dynamic Bitrate Change yes Forced constant QP yes Dynamic RC Mode Change no Subframe Readback yes Constrained Encoding no Intra Refresh yes Custom VBV Bufsize yes Dynamic Slice Mode yes Ref Pic Invalidiation yes PreProcess no Async Encoding yes Max MBs 262144 MAX MB per sec 983040 Lossless yes SAO yes Me Only Mode yes Lookahead yes AQ (temporal) no Weighted Prediction yes Max LTR Frames 7 10bit depth yes

So on my system the max MB count per second is restricted to 983040 which seems to be the same on your system by the error popup.
Hybrid calculates the MBs per secund through:

MAX MB per sec = (width / 16.0 + 0.5) * (height / 16.0 + 0.5) * fps

which would be in your case:
3840/16 * 2140/16 * 59.94 = 1 924 074
which is why Hybrid aborts here.
(983040 would be enough for 3840/16 * 2140/16 * 30 = 963000)

Grabbing the latest NVEncC 4.14 from the authors GoogleDrive, extracting the content of the 'NVEncC\64x'-folder into the 'Hybrid/64bit'-folder and renaming NVEncC64.exe to NVEncC.exe and running the call from above again doesn't change the output for me.

So there are multiple scenarios:
a. I/Hybrid calculates the macro blocks per second wrong
b. the value NVEncC output wrong data
->
a. if the calculation is wrong, I would need to know how to calculate this correctly (I don't see a mistake here since macro blocks should be 16x16 pixel in size)
b. If the restriction is really outdated, this needs to be fixed in NVEncC which Hybrid uses. Once this is fixed in NVEncC replacing the file like mentioned above and calling 'Config->Tools->Clear all cached tool infos' will fix the issue. In case this is a bug in NVEncCs output, I do not plan to write a workaround for this, since I believe this should be fixed in NVEncC then.

Cu Selur

Ps.: NVEncCs 'home' is https://github.com/rigaya/NVEnc