[HELP] after encoding ass into mp4, but there is no subtitle at all

[Selur]

select the filter the custom code should appear before, what does this mean?

You use Hybrids Vapoursynth support with a 'Filter Order' (see "Filtering->Vapoursynth->Misc->Filter Order/Queue"), 'Insert before' tells Hybrid where that it should insert the custom code you wrote before a specific filter.
Instead of 'Deinterlace' it would be more secure to select 'Adding Subtitle'.

Cu Selur