Selur's Little Message Board
[HELP] How to convert video to higher bit depth before running certain filter - Printable Version

+- Selur's Little Message Board (https://forum.selur.net)
+-- Forum: Hybrid - Support (https://forum.selur.net/forum-1.html)
+--- Forum: Problems & Questions (https://forum.selur.net/forum-3.html)
+--- Thread: [HELP] How to convert video to higher bit depth before running certain filter (/thread-4370.html)

How to convert video to higher bit depth before running certain filter - andrewschen - 18.04.2026

After SMdegrain, there are banding in the output video. (8bit bt709 progressive bluray video)

AI tells me I should convert the video to higher bit depth before SMdegrain.

I try to insert the following script to Vapoursynth - custom - Insert before [smdegrain]:

clip = core.resize.Point(clip, format=vs.YUV420P16)

but it doesn't change much in the result.

What's the correct method?

RE: How to convert video to higher bit depth before running certain filter - Selur - 18.04.2026

You can enable "Filtering->Vapoursynth->Misc->UI->Show 'Gimmick'-controls", this will show additional controls next to most of the filters.
[Image: grafik.png]
There you can set 'Restrict bit depth' to a bit depth.
Hybrid will then convert to the selected bit depth before applying the filter.
Assuming the filter you apply does not support the bit depth, Hybrid will convert down to a bit depth the filter does support.
Any denoising, degrain and similar filter that does smooth in some way or another can lead to banding when using low bit depth precision.
Note that, if your output is 8bit the output might still contain banding, even if your whole chain beforehand is in i.e. 32bit, unless you add debanding/dithering/noise to the video. Simply due to the calculation precision and what banding is and how happens.
see: https://en.wikipedia.org/wiki/Colour_banding

Quote:I try to insert the following script to Vapoursynth - custom - Insert before [smdegrain]:

clip = core.resize.Point(clip, format=vs.YUV420P16)
unless you let Hybrid know that you changed the color format (see tool-tip of 'Insert before'), this can cause issues in itself.

Cu Selur

RE: How to convert video to higher bit depth before running certain filter - andrewschen - 19.04.2026

I forgot to change my player's display output depth to 10-bit....no wonder it look so bad Shy 
Just by changing to right display setting, the situtation is much relieved.
But the giimmick is still very useful.... I didn't found out about that before. Thanks.

RE: How to convert video to higher bit depth before running certain filter - andrewschen - 23.04.2026

(18.04.2026, 18:56)Selur Wrote:
Quote:I try to insert the following script to Vapoursynth - custom - Insert before [smdegrain]:

clip = core.resize.Point(clip, format=vs.YUV420P16)
unless you let Hybrid know that you changed the color format (see tool-tip of 'Insert before'), this can cause issues in itself.

Cu Selur

Another question, would it be better to ONLY convert to high bit depth, without applying any other filter, before begin to encode? and how to do that in hybrid?

I choose "4 - nearest neighbour resize" in the main Crop/Resize tab, and insert the following script in 'Insert before' Resize:

Quote:clip = core.resize.Point(clip=clip, format=vs.YUV444P16, dither_type="error_diffusion")
- '# colorformat <YUV444P16>'


but in the final VS scripts still shows:

Quote:# adjusting output color from YUV420P8 to YUV420P10 for SvtAv1Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion")


RE: How to convert video to higher bit depth before running certain filter - Selur - 23.04.2026

- '# colorformat <YUV444P16>'
should be just:
# colorformat YUV44416
and since you configured SvtAv1Model for YUV420P10 output, Hybrid would add
# adjusting output color from YUV444P16 to YUV420P10 for SvtAv1Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion")

Quote: Another question, would it be better to ONLY convert to high bit depth, without applying any other filter, before begin to encode? and how to do that in hybrid?
No clue, why you would do that (makes no sense to me).
But, ... you could do it by adding a custom section with - assuming you want to use YUV444P16:
# requires colorformat YUV444P16
(nothing more, is needed)
Hybrid would then create something like:
...
# defining beforeDeCross-function - START
def beforeDeCross(clip):
  return [clip]
# defining beforeDeCross-function - END
...
[clip] = beforeDeCross(clip)
# clip current meta; color space: YUV444P16, bit depth: 16, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive, full height: true
# adjusting output color from YUV444P16 to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10)

side note: Hybrid will use the dither_type configured under "Filtering->Vapoursynth->Misc->Script->Dither durin color change", when converting from a lower to a higher
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion")

Behind each custom section Hybrid will add a comment, which reports the characteristics it assumes. Smile

Hybrid will adjust the output of the script to whatever color format is configured in the encoder.

Cu Selur

Ps.: I think there is a bug somewhere, Hybrid should only set the dithering when going from a higher to lower bit depth. => uploaded a new dev which fixes that.

RE: How to convert video to higher bit depth before running certain filter - andrewschen - 24.04.2026

I'm just curious about how things work, experimenting....  Blush

BTW If put # requires colorformat YUV444P10 in custom script, the result shows
Quote:# adjusting output color from YUV444P10 to YUV420P10 for SvtAv1Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_in_s="limited")

If it's # requires colorformat YUV444P12:
Quote:# adjusting output color from YUV444P12 to YUV420P10 for SvtAv1Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion") 

and if it's # requires colorformat YUV444P16:
Quote:# adjusting output color from YUV444P16 to YUV420P10 for SvtAv1Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10)
why there's no dithering if it's 16bit->10bit? because it's already too good?

RE: How to convert video to higher bit depth before running certain filter - Selur - 24.04.2026

No, dithering should be there, whenever going from a higher to a lower bit depth.
Did you use the latest dev?

RE: How to convert video to higher bit depth before running certain filter - Selur - 24.04.2026

Yes, should be fixed in latest dev, at least I get:
# clip current meta; color space: YUV444P16, bit depth: 16, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive, full height: true
# adjusting output color from YUV444P16 to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion")

Cu Selur

RE: How to convert video to higher bit depth before running certain filter - andrewschen - 24.04.2026

Yes, just dl & tested, fixed in the latest dev, it now show


clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion")