[HELP] How do I keep DVD rips small in size?

[Selur]

I'd wish to have them smaller in size without losing quality.

Not possible with any format I know of. All current formats like H.264, H.265, AV-1, VP9 are all will either:
a. be lossless and produce huge files
or
b. be lossy
Your best bet is get something where you don't see the difference.
-> try for example H.264 with x264 as encoder with a crf of 16 or H.265 with x265 as encoder and crf 16.

-----------
To answer you title: 'How do I keep DVD rips small in size?'
a. you filter your input to remove unneeded details, noise and artifacts (yes, DVDs are full of compression artifacts).
b. you recompress into a format which produces a smaller file size that produces for you perceivable quality loss.
No there are no clever/smart/magic ways to do this always perfectly.

-------------
also: mkv is a container format like vob/mpeg/ts/m2ts/mp4
MakeMKV is a multiplexer which repacks for example DVDs or Blu-rays (or more precise parts of them) into mkv.

H.264, H.265 are MPEG video compression formats.
mp3 and aac are MPEG audio compression formats.

On the other hand, maybe there exist "smart" video file formats that compress the video or something to make it smaller.

I know of no lossless transcoder from MPEG-2 (which is the video format inside a vob container on a DVD) to any format that compresses better.

Maybe Hybrid could make a change, but that programme is pretty advanced so I wouldn't really know what settings to go with.

Yes, Hybrid is probably not the right tool for you. StaxRip, Handbrake, MeGUI, VidCoder,.. are probably more suited for you.
(You are welcome to try, but Hybrid in general is aimed for more advanced users who seek more control and screws to turn, but know what they are doing. )

Cu Selur