4K transcoding problem with NVENC (outdated false limitation in Hybrid)

[No limit]

What does:

NVEncC --check-features

output on your system?

I get for example:

Environment Info OS : Windows 10 x64 (17134) CPU: AMD Ryzen 7 1800X Eight-Core Processor (8C/16T) RAM: Used 5469 MB, Total 32696 MB GPU: GeForce GTX 1070 Ti (19 EU) @ 1683 MHz (399.07) List of available features. Codec: H.264/AVC Max Bframes               4 B Ref Mode                yes RC Modes                  63 Field Encoding            yes MonoChrome                no FMO                       no Quater-Pel MV             yes B Direct Mode             yes CABAC                     yes Adaptive Transform        yes Max Temporal Layers       0 Hierarchial P Frames      no Hierarchial B Frames      no Max Level                 51 Min Level                 1 4:4:4                     yes Max Width                 4096 Max Height                4096 Dynamic Resolution Change yes Dynamic Bitrate Change    yes Forced constant QP        yes Dynamic RC Mode Change    no Subframe Readback         yes Constrained Encoding      yes Intra Refresh             yes Custom VBV Bufsize        yes Dynamic Slice Mode        yes Ref Pic Invalidiation     yes PreProcess                no Async Encoding            yes Max MBs                   65536 MAX MB per sec            983040 Lossless                  yes SAO                       no Me Only Mode              yes Lookahead                 yes AQ (temporal)             yes Weighted Prediction       yes Max LTR Frames            8 10bit depth               no Codec: H.265/HEVC Max Bframes               0 RC Modes                  63 Field Encoding            no MonoChrome                no Quater-Pel MV             yes B Direct Mode             no Max Temporal Layers       0 Hierarchial P Frames      no Hierarchial B Frames      no Max Level                 62 Min Level                 1 4:4:4                     yes Max Width                 8192 Max Height                8192 Dynamic Resolution Change yes Dynamic Bitrate Change    yes Forced constant QP        yes Dynamic RC Mode Change    no Subframe Readback         yes Constrained Encoding      no Intra Refresh             yes Custom VBV Bufsize        yes Dynamic Slice Mode        yes Ref Pic Invalidiation     yes PreProcess                no Async Encoding            yes Max MBs                   262144 MAX MB per sec            983040 Lossless                  yes SAO                       yes Me Only Mode              yes Lookahead                 yes AQ (temporal)             no Weighted Prediction       yes Max LTR Frames            7 10bit depth               yes

So on my system the max MB count per second is restricted to 983040 which seems to be the same on your system by the error popup.
Hybrid calculates the MBs per secund through:

MAX MB per sec = (width / 16.0 + 0.5) * (height / 16.0 + 0.5) * fps

which would be in your case:
3840/16 * 2140/16 * 59.94 = 1 924 074
which is why Hybrid aborts here.

Grabbing the latest NVEncC 4.14 from the authors GoogleDrive, extracting the content of the 'NVEncC\64x'-folder into the 'Hybrid/64bit'-folder and renaming NVEncC64.exe to NVEncC.exe and running the call from above again doesn't change the output for me.

So there are multiple scenarios:
a. I/Hybrid calculates the macro blocks per second wrong
b. the value NVEncC output wrong data
->
a. if the calculation is wrong, I would need to know how to calculate this correctly (I don't see a mistake here since macro blocks should be 16x16 pixel in size)
b. If the restriction is really outdated, this needs to be fixed in NVEncC which Hybrid uses. Once this is fixed in NVEncC replacing the file like mentioned above and calling 'Config->Tools->Clear all cached tool infos' will fix the issue.

Cu Selur

That's the problem with the over automatizing to prevent every possible failure, and the software won't start correctly. Can you switch off this feature for GF 1000 series, because with the new drivers these cards had not such problems during 4K transcoding, which means that the calculation is somehow wrong in the Hybrids error-message. 

Meanwhile I tried 8K NVENC transcoding in Staxrip, and everything works fine. GF 1000 series with Pascal support have a max encoding limit in 8K resolution. So in the case of Pascal driven Cards, you can turn off these thing. Okay, people who use old VGA drivers can have problems, but new drivers fixed that Issue from 2018.

Here is the NVENC support matrix (Encode decode everything.)  https://developer.nvidia.com/video-encod...ort-matrix

And it has right, 8K encoding works with Pascal engines.