[HELP] New to Hybrid, trying to figure some things out...

[harrisonjr98]

Hello all! I am new to Hybrid (and VirtualDub, for that matter) but have recently completed the capture process of a box of old home video Video8/Hi8 tapes. I captured into VirtualDub (720x480 @ 29.97 interlaced, YUV 4:2:2 8-bit, PCM audio) and I am trying to use Hybrid to deinterlace with QTGMC and then apply some basic processing to clean it up. There is a good amount of chroma noise (and just general noise/grain) happening on a lot of the footage, especially the parts that were shot in suboptimal lighting conditions (of course.)

I have about 20 hours of total footage that I want to de-interlace, filter/process and encode to H.264 in an mp4 container at a reasonable file size/bitrate for easy sharing with the family over Google Drive.

I figured that the best way to learn would be to immerse myself in the settings for a few hours and play around with them, checking in with old forum threads via Google Search when I really got stumped. I came to realize a few things, and I want to make sure they're not flawed logic:

1) upscaling to 1440x1080 (input PAR 10:11 and output PAR convert to square pixels) as opposed to the 720x540 that I was originally planning on seemed to produce much higher quality results from the VapourSynth filters that I ended up using, namely MLDegrain and MC Temporal Denoise.

2) Using QTGMC on Fast or Faster setting seemed to minimize its own denoise/degrain filtering, and I got seemingly better results using QTGMC on Fast/Faster + MLDegrain, MC Temporal Denoise, CAS Sharpen than I did using QTGMC on a slower setting that appears to try and accomplish more of this itself.

3) I started out by using specific bitrate (2-pass) but after doing some online research, realized that CRF (1-pass, QF set to 18) is probably what I am actually after. I am trying to keep the file sizes manageable in order to host 20+ hours of video in cloud storage indefinitely for family access, but I don't want to sacrifice the bitrate so much that the end-product looks like junk.

4) I tried a LOT of settings to reduce noise/grain without making everyone look too plastic-y or over-smoothed, and ultimately I'm still not sure if I'm satisfied. Part of me thinks that playing back the interlaced HuffYUV capture looks a good amount better/sharper than the deinterlaced version, albeit annoying to watch on a progressive display. To try and restore some of this sharpness, I ended up using CAS (set to 0.7) and then GrainFactory 3 to reintroduce a moderately sized grain which made things look more "correct" to my brain, even though I know it is trying to emulate a filmic grain and doesn't *really* make sense to add over this type of footage. Unfortunately, using the default grain size settings, this ballooned my file sizes (still CRF, 18) by a gargantuan amount - over 3x!

So... is there a better way to do that? The logic behind my current settings has been:

-Capture file is as raw as possible.
-Deinterlace with QTGMC, Fast/Faster settings to avoid it doing too much processing on its own
-Degrain with MLDegrain (strength 3), denoise with MC Temporal Denoise (medium)- I messed with the intensity of both of these and found this to be a good balance.
-Re-sharpen with CAS (0.7) - I kept going between 0.5, 0.6 and 0.7 but decided on the latter.
-Reintroduce fake grain/noise with GrainFactory to visually make it look more like a natively "soft" picture from a vintage format again as opposed to a smoothed-over digitized plastic-fest.
-Export H.264 in .mp4 container using CRF (QF 18), 1440x1080, 59.94fps deinterlaced for an ideal modern, shareable final file.

The only issue I am having is that adding back grain is ballooning my file sizes when set to CRF 18, and setting them instead to constant bitrate 2-pass - even if I choose a setting like 14000 kbps that should be high enough - introduces very obvious compression artifacts on many scenes, presumably because the bitrate is still not high enough to make heads or tails of the small grain. If I turn off GrainFactory, I have small file sizes and ample detail but I cannot shake the feeling that the end product looks extremely processed/fake. If I turn off GrainFactory *and* reduce the strength of MLDegrain and MLTempNoise, I have a lot of chroma noise and visual evidence of the tape degradation.

So I am trying to figure out what I need to change about my current settings (if I am even using the best filters for my project to begin with) to yield ideal file sizes, and moderate denoise/degrain without looking silly.

Another problem I ran into this morning is that whenever I select any of the De-Halo filters, something breaks in the script and I can't run them - it says "expected 'else' after 'if' expression (dehalo.py)". There is some serious Haloing in the footage that will need to be addressed, but I didn't get into troubleshooting this earlier.

Anyway, I know that this type of filtering is not a one-size fits all, but I am looking for settings that I can broadly apply to the entire series of tapes. I will of course be hanging onto the HuffYUV avi "masters" in case I want to revisit this years down the road and try to get a better result.

Thanks in advance for any help/assistance you can provide!

[Selur]

Another problem I ran into this morning is that whenever I select any of the De-Halo filters, something breaks in the script and I can't run them - it says "expected 'else' after 'if' expression (dehalo.py)". There is some serious Haloing in the footage that will need to be addressed, but I didn't get into troubleshooting this earlier.

Replace dehalo.py with https://github.com/Selur/VapoursynthScri.../dehalo.py

The general steps are correct.
MLDegrain, MC Temporal Denoise are rather heavy hitters, depending on the source using for example SMDegrain with higher SAD/-C might work too.
Would need a short sample of the source to suggest something. Depending on your setup using machine learning based filters might also be possible.

Cu Selur

[harrisonjr98]

Hmm, I tried replacing dehalo.py and I also updated to the latest Hybrid version that was released yesterday. Now it seems like I cannot use some of my other filters for whatever reason. Yesterday I was able to use everything that I tried to use, except the De-Halo stuff.




Today I am getting this error when I try to use MLDegrain:

2025-05-18 10:12:22.616
VapourSynth plugins manager: Failed to load vapoursynth library!
Please set up the library search paths in settings.
Failed to evaluate the script:
Python exception: name 'scale' is not defined
Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3387, in vapoursynth._vpy_evaluate
File "src\\cython\\vapoursynth.pyx", line 3388, in vapoursynth._vpy_evaluate
File "C:\Users\harri\AppData\Local\Temp\tempPreviewVapoursynthFile10_12_22_522.vpy", line 53, in
clip = degrain.MLDegrain(clip=clip, soft=[0,0,0])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\vsscripts\degrain.py", line 334, in MLDegrain
lev2 = MLD_helper(sm2, sm2, tr, thSAD, rec, chroma, soft[0]) # Filter on smalle scale
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\vsscripts\degrain.py", line 368, in MLD_helper
sup1 = S(DitherLumaRebuild(srch, 1), hpad=bs, vpad=bs, pel=pel, sharp=1, rfilter=4)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\vsscripts\degrain.py", line 840, in DitherLumaRebuild
t = f'x {scale(16, bits)} - {scale(219, bits)} / 0 max 1 min' if is_integer else 'x 0 max 1 min'
^^^^^
NameError: name 'scale' is not defined


Trying to use Dehalo Alpha with the NEW dehalo.py in the \vsscripts directory:

2025-05-18 10:14:18.222
VapourSynth plugins manager: Failed to load vapoursynth library!
Please set up the library search paths in settings.
Failed to evaluate the script:
Python exception: name 'TypeVar' is not defined
Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3387, in vapoursynth._vpy_evaluate
File "src\\cython\\vapoursynth.pyx", line 3388, in vapoursynth._vpy_evaluate
File "C:\Users\harri\AppData\Local\Temp\tempPreviewVapoursynthFile10_14_18_016.vpy", line 27, in
import dehalo
File "C:\Program Files\Hybrid\64bit\vsscripts\dehalo.py", line 5, in
from typing import Union, Optional, Sequence; TypeVar
^^^^^^^
NameError: name 'TypeVar' is not defined


CAS and MCTemporalDenoise are still able to generate previews for me, but I am very briefly seeing the script failed pop-up with this message before the preview window comes up:

2025-05-18 10:16:04.041
VapourSynth plugins manager: Failed to load vapoursynth library!
Please set up the library search paths in settings.
2025-05-18 10:16:05.043
Script was successfully evaluated. Output video info:
Frames: 10864 | Time: 0:03:01.248 | Size: 720x480 | FPS: 60000/1001 = 59.9401 | Format: YUV420P8
2025-05-18 10:16:05.523
Qt warning: QPixmap:caled: Pixmap is a null pixmap


It seems like something got messed up in my Hybrid install with the update. I tried to uninstall and re-download the latest version for a clean restart, and I am still seeing this same issue.



Regardless, here are a couple of samples of the source that I am working with. https://we.tl/t-vbvOxHjxIc

One clip is the raw capture huffyuv, the other clip was the "best" result that I got after tinkering for hours with Hybrid yesterday. I think I ended up using MLDegrain, MC Temporal Denoise, CAS, and GrainFactory on this clip, but it was before I switched to CRF encoding so there is blocky compression instead of the grain that I was going for.

This clip is very indicative of how most of these tapes look - one well-lit daytime scene followed by a dark/underexposed one with a lot of noise, followed by another well-lit one etc etc. I didn't want to mess with the proc-amp settings on my GV-USB2 capture device without knowing what I was doing, so there are definitely some clipped whites and blacks, but overall my capture looked fairly close to what I was seeing on the camcorder's LCD as the tapes played back.

Overall, I dislike how my processing attempts are turning out compared to how "natural" the source capture looks to me. The edges/lines are too "hard" in the processed version, but if I strip away all of the vapoursynth stuff and just have QTGMC deinterlace on Fast/Faster as my only processing, that output has far too much chroma noise for my liking, especially on the darker scenes.

As I said, I am brand new to all of this, so I may not even possess the language yet to describe what I am actually looking for here, but if I had to describe it in layman's terms I would say something like:

I want my final product to appear as noise-reduced footage that doesn't look overly smoothed, *especially* around the edges of people and objects where the lines appear 'thicker' for whatever reason. I just want a reasonable balance between processing, render time, and settings that are universal enough for me to broadly apply across most of these tapes with minimal tweaking. Maybe I'm asking for the world, maybe I'm asking for something super simple - I don't know!

Thanks again for taking the time to help people out directly in this forum. It means the world.

[Selur]

from typing import Union, Optional, Sequence; TypeVar

damn a typo,.. ; should be ,

will look at it

[Selur]

CAS and MCTemporalDenoise are still able to generate previews for me, but I am very briefly seeing the script failed pop-up with this message before the preview window comes up:

2025-05-18 10:16:04.041
VapourSynth plugins manager: Failed to load vapoursynth library!
Please set up the library search paths in settings.
2025-05-18 10:16:05.043
Script was successfully evaluated. Output video info:
Frames: 10864 | Time: 0:03:01.248 | Size: 720x480 | FPS: 60000/1001 = 59.9401 | Format: YUV420P8

can't reproduce that.

Updated:
https://github.com/Selur/VapoursynthScri.../dehalo.py
and
https://github.com/Selur/VapoursynthScri...degrain.py
and
https://github.com/Selur/VapoursynthScri...esample.py

Let me know whether replacing those fixes the dehalo&mldegrain problems for you.

I would use something like:

# Deinterlacing using QTGMC
clip = qtgmc.QTGMC(Input=clip, Preset="Fast", TFF=True, opencl=True) # new fps: 59.94
# Making sure content is preceived as frame based
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive
# stabilizing using Stab
clip = stabilize.Stab(clp=clip,range=2,mirror=0)
clip = core.std.Crop(clip=clip, left=10, right=8, top=2, bottom=6)# cropping to 702x472
# removing grain using SMDegrain
clip = smdegrain.SMDegrain(input=clip, contrasharp=10, interlaced=False, opencl=True, device=-1)

as a general start and then add additional filtering on the evening scenes.

Cu Selur

Ps.: Those thick lines are dark halos.

[Selur]

=> I'll upload a new Hybrid version in a bit.

[harrisonjr98]

Ah interesting, so you're thinking I maybe have no need for Temporal Denoise, CAS, or De-Halo based on my source footage? I would definitely love to keep it simple if I can get an output I'm happy with 

I will look out for that new Hybrid version shortly and give this a shot.

[Selur]

About the dark halos I would try aWarpSharp2 (depth maybe 16) on 'luma only' with a Sobel EdgeMask.
New version should be up in ~7min
Temporal Denoise seems to be to heavy filtering for the bright scene, when using CAS may be stick to 0.5 or 0.6 since it is contrast enhancing, thus black lines will 'pop' more.
For the might brighter halos I would try YAHR.
(best would be to avoid those during capturing, since they are usually caused by some sharpening function)

Cu Selur

Ps.: What graphic card are you using?
PPs.: If you smooth too much, adding some grain at the end can sometimes help.

[Selur]

In the bright part of the video denoising, dehalo while keeping fine details is problematic.
The low light part is the more problematic since details&noise are practically one.
Maybe something like:
script: https://pastebin.com/avvwAY28 (only stronger degrain, thining dark lines and some sharpening + color temperature change for the second half to lessen the yellow tone)
clip: https://www.mediafire.com/file/yal6w69s9...1.mp4/file

Cu Selur