4K transcoding problem with NVENC (outdated false limitation in Hybrid)

[No limit]

What does:

NVEncC --check-features

output on your system?

I get for example:

Environment Info
OS : Windows 10 x64 (17134)
CPU: AMD Ryzen 7 1800X Eight-Core Processor (8C/16T)
RAM: Used 5469 MB, Total 32696 MB
GPU: GeForce GTX 1070 Ti (19 EU) @ 1683 MHz (399.07)

List of available features.
Codec: H.264/AVC
Max Bframes               4
B Ref Mode                yes
RC Modes                  63
Field Encoding            yes
MonoChrome                no
FMO                       no
Quater-Pel MV             yes
B Direct Mode             yes
CABAC                     yes
Adaptive Transform        yes
Max Temporal Layers       0
Hierarchial P Frames      no
Hierarchial B Frames      no
Max Level                 51
Min Level                 1
4:4:4                     yes
Max Width                 4096
Max Height                4096
Dynamic Resolution Change yes
Dynamic Bitrate Change    yes
Forced constant QP        yes
Dynamic RC Mode Change    no
Subframe Readback         yes
Constrained Encoding      yes
Intra Refresh             yes
Custom VBV Bufsize        yes
Dynamic Slice Mode        yes
Ref Pic Invalidiation     yes
PreProcess                no
Async Encoding            yes
Max MBs                   65536
MAX MB per sec            983040
Lossless                  yes
SAO                       no
Me Only Mode              yes
Lookahead                 yes
AQ (temporal)             yes
Weighted Prediction       yes
Max LTR Frames            8
10bit depth               no

Codec: H.265/HEVC
Max Bframes               0
RC Modes                  63
Field Encoding            no
MonoChrome                no
Quater-Pel MV             yes
B Direct Mode             no
Max Temporal Layers       0
Hierarchial P Frames      no
Hierarchial B Frames      no
Max Level                 62
Min Level                 1
4:4:4                     yes
Max Width                 8192
Max Height                8192
Dynamic Resolution Change yes
Dynamic Bitrate Change    yes
Forced constant QP        yes
Dynamic RC Mode Change    no
Subframe Readback         yes
Constrained Encoding      no
Intra Refresh             yes
Custom VBV Bufsize        yes
Dynamic Slice Mode        yes
Ref Pic Invalidiation     yes
PreProcess                no
Async Encoding            yes
Max MBs                   262144
MAX MB per sec            983040
Lossless                  yes
SAO                       yes
Me Only Mode              yes
Lookahead                 yes
AQ (temporal)             no
Weighted Prediction       yes
Max LTR Frames            7
10bit depth               yes

So on my system the max MB count per second is restricted to 983040 which seems to be the same on your system by the error popup.
Hybrid calculates the MBs per secund through:

MAX MB per sec = (width / 16.0 + 0.5) * (height / 16.0 + 0.5) * fps

which would be in your case:
3840/16 * 2140/16 * 59.94 = 1 924 074
which is why Hybrid aborts here.

Grabbing the latest NVEncC 4.14 from the authors GoogleDrive, extracting the content of the 'NVEncC\64x'-folder into the 'Hybrid/64bit'-folder and renaming NVEncC64.exe to NVEncC.exe and running the call from above again doesn't change the output for me.

So there are multiple scenarios:
a. I/Hybrid calculates the macro blocks per second wrong
b. the value NVEncC output wrong data
->
a. if the calculation is wrong, I would need to know how to calculate this correctly (I don't see a mistake here since macro blocks should be 16x16 pixel in size)
b. If the restriction is really outdated, this needs to be fixed in NVEncC which Hybrid uses. Once this is fixed in NVEncC replacing the file like mentioned above and calling 'Config->Tools->Clear all cached tool infos' will fix the issue.

Cu Selur

That's the problem with the over automatizing to prevent every possible failure, and the software won't start correctly. Can you switch off this feature for GF 1000 series, because with the new drivers these cards had not such problems during 4K transcoding, which means that the calculation is somehow wrong in the Hybrids error-message. 

Meanwhile I tried 8K NVENC transcoding in Staxrip, and everything works fine. GF 1000 series with Pascal support have a max encoding limit in 8K resolution. So in the case of Pascal driven Cards, you can turn off these thing. Okay, people who use old VGA drivers can have problems, but new drivers fixed that Issue from 2018.

Here is the NVENC support matrix (Encode decode everything.)  https://developer.nvidia.com/video-encod...ort-matrix

And it has right, 8K encoding works with Pascal engines.