[BUG] Problem with QTGMC + EdiExt + Def + more

[Selur]

I am trying to replicate this flow in hybrid, but I cannot figure out how to generate the VFR times and then insert them into the final container.

You can not.
Hybrid does not support deinterlacing VFR content while keeping it vfr.
This is, because:

• I think it's total madness to create such content.
  
• the only cases where I have seen such content is when MakeMKV created it, out of DVDs or Blu-rays. But instead of VFR and interlaced the content should have been flagged CFR and telecined.
  
• simple extracting and reapplying the time codes isn't possible, since deinterlacing and IVTC can change the frame count!
  

=> I suspect your content isn't really interlaced and vfr to begin with. Judging by the name of your input file, your content should not be vfr, since Blu-rays

"TFM + TDecimate → CFR + Timecodes"

Hybrid doesn't support mode 3 or 5 for TDecimate, also those modes are not meant to be applied on VFR content, but on CFR content.

Side note: those steps seem like something some ai chat bot came up with,...

Apply AI (DEF / vs_deepdeinterlace)

Use the AI model to remove ghosting, shimmer, and improve frame quality.
This is applied on the CFR clip generated from step 1.

Hybrid also does not support applying DeF, which is for deinterlacing, on progressive content. (after step 1. the content should be progressive)
Also from my experience DeF does not really remove ghosting, it can improve some of the shimmering (but that could be addressed better) and it might improve the quality of the deinterlacing compared to some deinterlacers on some content.

About the crash of the preview: Sadly, I can't reproduce the issue.
using the same QTGMC settings you did:

# Deinterlacing using QTGMC
clipEdiExt = clip
# adjusting color space from YUV420P8 to RGBS for vsDDD
clipEdiExt = core.resize.Bicubic(clip=clipEdiExt, format=vs.RGBS, matrix_in_s="470bg", range_in_s="limited", range_s="full")
clipEdiExt = vs_deepdeinterlace.DeF(clip=clipEdiExt, tff=False, fp16=True, device="cuda")
# Making sure content is preceived as frame based
clipEdiExt = core.std.SetFrameProps(clip=clipEdiExt, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive
clipEdiExt = core.resize.Bicubic(clip=clipEdiExt, format=vs.YUV420P8, matrix_s="470bg", range_in_s="full", range_s="limited", dither_type="error_diffusion") # additional resize to allow target color sampling
clip = qtgmc.QTGMC(Input=clip, Preset="Slower", InputType=0, TFF=False, TR2=1, SourceMatch=3, Lossless=1, MatchPreset="Slower", MatchPreset2="Slower", opencl=True, EdiExt=clipEdiExt) # new fps: 29.97
# Making sure content is preceived as frame based
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive
clip = clip[::2] # selecting previously even frames

works fine here.
I assume you have a GPU with at least 16GB VRAM and supports FP16, since when I apply Def in 1920x1080 even with FP16 (QTGMC always does this when DeF is used) uses all the 16 GB VRAM of my GeForce RTX 4080 (and takes ages).
(also eats a lot of normal RAM, around ~30GB peeking to around 43GB)

=> If you can share a sample of your source (i.e. by splitting it using mkvtoolnix) I can look at that sample and
a. try to reproduce the problem
b. try to suggest something.

Cu Selur

Ps.: Playing around, I could reproduce the crash here, when my system ran out of vram, so I guess that is the same issue you are having. (to be frank applying DeF on HD content seems simply not really feasible and worth it)