Hybrid ignore settings

[Selur]

Looking at the encoding call:

"C:\Program Files\Hybrid\64bit\ffmpeg.exe" -y -loglevel fatal -noautorotate -nostdin -threads 8 -ignore_editlist true -i "G:\3D\Tuts\Blender\Sculpting\HUMAN\HUMAN_course-videos\HUMAN_chapter_02\C02L00_intro-to-modeling-and-detailing.mp4" -map 0:0 -an -sn -vf scale=in_range=tv:out_range=tv -pix_fmt yuv420p10le -strict -1 -vsync 0 -f yuv4mpegpipe - | "C:\Program Files\Hybrid\64bit\NVEncC.exe" --y4m -i - --fps 60.000 --codec h265 --profile main10 --level auto --tier high --sar 1:1 --lookahead 32 --output-depth 10 --multipass "2pass-quarter" --vbr 100 --max-bitrate 800000 --gop-len 600 --strict-gop --ref 3 --weightp --bframes 3 --bref-mode middle --no-b-adapt --mv-precision Q-pel --cu-max 32 --cu-min 8 --preset default --colorrange limited --colormatrix bt2020nc --cuda-schedule auto --psnr --ssim --output "C:\Users\Lem\AppData\Local\Temp\C02L00_intro-to-modeling-and-detailing_1_2023-05-08@08_26_25_9710_02.265"

Is "--vbr 100" what you wanted? 100kbit/s seems way too low for 1920x1080.

That said, NVEncC starts with those settings:

OS Version     Windows 10 x64 (19044) [UTF-8]
CPU            AMD Ryzen 5 5600X 6-Core Processor [4.67GHz] (6C/12T)
GPU            #0: NVIDIA GeForce GTX 1650 SUPER (1280 cores, 1725 MHz)[PCIe3x16][531.79]
NVENC / CUDA   NVENC API 12.0, CUDA 12.1, schedule mode: auto
Input Buffers  CUDA, 44 frames
Input Info     y4m(yv12(10bit))->p010 [AVX2], 1920x1080, 60/1 fps
Vpp Filters    copyHtoD
               ssim psnr (yv12(10bit))
Output Info    H.265/HEVC main10 @ Level auto
               1920x1080p 1:1 60.000fps (60/1fps)
Encoder Preset default
Rate Control   VBR
Multipass      2pass-quarter
Bitrate        100 kbps (Max: 800000 kbps)
Target Quality auto
Initial QP     I:20  P:23  B:25
QP Offset      cb:0  cr:0
VBV buf size   auto
Lookahead      on, 32 frames, Adaptive I Insert
GOP length     600 frames
B frames       3 frames [ref mode: middle]
Ref frames     3 frames, MultiRef L0:auto L1:auto
AQ             off
CU max / min   32 / 8
VUI            matrix:bt2020nc,range:limited
Others         mv:Q-pel
64 frames: 99.22 fps, 17037 kb/s, GPU 3%, VE 0%                                
193 frames: 133.38 fps, 12354 kb/s, GPU 20%, VE 99%, VD 10%                    
322 frames: 143.17 fps, 11408 kb/s, GPU 20%, VE 99%, VD 10%                    
451 frames: 147.72 fps, 11188 kb/s, GPU 20%, VE 100%, VD 10%                  
581 frames: 150.64 fps, 10081 kb/s, GPU 20%, VE 99%, VD 10%                    
711 frames: 152.48 fps, 8757 kb/s, GPU 20%, VE 99%, VD 10%                    
841 frames: 153.83 fps, 8070 kb/s, GPU 20%, VE 99%, VD 10%                    
970 frames: 154.75 fps, 8764 kb/s, GPU 20%, VE 99%, VD 11%                    
1099 frames: 155.38 fps, 9056 kb/s, GPU 20%, VE 99%, VD 11%                    
1228 frames: 155.92 fps, 9305 kb/s, GPU 20%, VE 100%, VD 10%                  
1357 frames: 156.37 fps, 9316 kb/s, GPU 20%, VE 99%, VD 10%                    
1486 frames: 156.75 fps, 9561 kb/s, GPU 20%, VE 99%, VD 10%                    
1615 frames: 157.04 fps, 9721 kb/s, GPU 20%, VE 100%, VD 10%                  
1745 frames: 157.38 fps, 10027 kb/s, GPU 20%, VE 100%, VD 11%                  
on_stopAllPushButton_clicked

but uses really high bit rates.
You can try whether it helps you update the NVEncC.exe with the latest from https://github.com/rigaya/NVEnc/releases (use NVCEncC64.exe and rename it)
The calls itself seem to be okay.

Cu Selur