Hybrid is not using 100% CPU

[Selur]

test.vpy threw an error that it could not open the index file

Then you either didn't check the Vapoursynth Preview, which creates the index file, or you closed Hybrid, which deletes the index file.

Also,
unrelated, even though I set --psy-rd=2 in Hybrid the output was 0.

psy-rd=2 is the default.

--[no-]psy-rd <0..5.0>        Strength of psycho-visual rate distortion optimization, 0 to disable. Default 2.0

source: x265 --fullhelp
which is why it's not shown in the command line, while 'x265->Misc->Random Stuff->Minimize command line' is shown.
This seems like a bug in x265,since it does not seem to set psy-rd=2.00 when it's not set explicitly in the command line.
-> I'll write a workaround for it. (as a workaround, you can disable 'Minimize command line')

Cu Selur