Field Order

[Selur]

If the field order of the source is wrong of the source has no field order flag (this is not inherent to lossless encoding), then you should use 'Overwrite input scan type to", to let Hybrid know what the correct scan order is for your source.
You normally should not need to change any other scan order related settings.

Read [INFO] Interlacing patterns for a basic understanding on how to figure out what the correct field order is.

Like written in the link above, if your source is progressive and progressively flagged, if you open the Vapoursynth Preview and navigate through your source you should not see any combing artifacts.

Jagged edges can happen, if:
a. your source not interlaced (put progressive or telecined)
b. the deinterlacing settings are not correct for your source, sometimes they need adjustment.
c. there is a problem with the source
d. the deinterlacer can't properly deinterlace everything

=> figure out if your source is interlaced, telecine, field-shifted, mixed or something similar.
If the source is not interlaced, but something else, don't blindly deinterlace.
If it's interlaced, try tweaking the deinterlacing settings.
If that is not enough, try cleaning up after the deinterlacer by using:
a. QTGMC in the Denoise-tab its main purpose is to handle bad deinterlaced content
b. try using Vinverse/Vinverse2 to remove issues
c. try using an anti-aliang filter.

No clue what you are doing or what you are doing wrong.
I would need a short sample, with horizontal movement, of the content you try to convert to look at it and to give specific advice.

Cu Selur