[HELP] How to convert video to higher bit depth before running certain filter

[andrewschen]

After SMdegrain, there are banding in the output video. (8bit bt709 progressive bluray video)

AI tells me I should convert the video to higher bit depth before SMdegrain.

I try to insert the following script to Vapoursynth - custom - Insert before [smdegrain]:

clip = core.resize.Point(clip, format=vs.YUV420P16)

but it doesn't change much in the result.

What's the correct method?

[Selur]

You can enable "Filtering->Vapoursynth->Misc->UI->Show 'Gimmick'-controls", this will show additional controls next to most of the filters.

There you can set 'Restrict bit depth' to a bit depth.
Hybrid will then convert to the selected bit depth before applying the filter.
Assuming the filter you apply does not support the bit depth, Hybrid will convert down to a bit depth the filter does support.
Any denoising, degrain and similar filter that does smooth in some way or another can lead to banding when using low bit depth precision.
Note that, if your output is 8bit the output might still contain banding, even if your whole chain beforehand is in i.e. 32bit, unless you add debanding/dithering/noise to the video. Simply due to the calculation precision and what banding is and how happens.
see: https://en.wikipedia.org/wiki/Colour_banding

I try to insert the following script to Vapoursynth - custom - Insert before [smdegrain]:

clip = core.resize.Point(clip, format=vs.YUV420P16)

unless you let Hybrid know that you changed the color format (see tool-tip of 'Insert before'), this can cause issues in itself.

Cu Selur

[andrewschen]

I forgot to change my player's display output depth to 10-bit....no wonder it look so bad  
Just by changing to right display setting, the situtation is much relieved.
But the giimmick is still very useful.... I didn't found out about that before. Thanks.

[andrewschen]

I try to insert the following script to Vapoursynth - custom - Insert before [smdegrain]:

clip = core.resize.Point(clip, format=vs.YUV420P16)

unless you let Hybrid know that you changed the color format (see tool-tip of 'Insert before'), this can cause issues in itself.

Cu Selur

Another question, would it be better to ONLY convert to high bit depth, without applying any other filter, before begin to encode? and how to do that in hybrid?

I choose "4 - nearest neighbour resize" in the main Crop/Resize tab, and insert the following script in 'Insert before' Resize:

clip = core.resize.Point(clip=clip, format=vs.YUV444P16, dither_type="error_diffusion")
- '# colorformat <YUV444P16>'

but in the final VS scripts still shows:

# adjusting output color from YUV420P8 to YUV420P10 for SvtAv1Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion")

[Selur]

- '# colorformat <YUV444P16>'

should be just:

# colorformat YUV44416

and since you configured SvtAv1Model for YUV420P10 output, Hybrid would add

# adjusting output color from YUV444P16 to YUV420P10 for SvtAv1Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion")

Another question, would it be better to ONLY convert to high bit depth, without applying any other filter, before begin to encode? and how to do that in hybrid?

No clue, why you would do that (makes no sense to me).
But, ... you could do it by adding a custom section with - assuming you want to use YUV444P16:

# requires colorformat YUV444P16

(nothing more, is needed)
Hybrid would then create something like:

...
# defining beforeDeCross-function - START
def beforeDeCross(clip):
  return [clip]
# defining beforeDeCross-function - END
...
[clip] = beforeDeCross(clip)
# clip current meta; color space: YUV444P16, bit depth: 16, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive, full height: true
# adjusting output color from YUV444P16 to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10)

side note: Hybrid will use the dither_type configured under "Filtering->Vapoursynth->Misc->Script->Dither durin color change", when converting from a lower to a higher
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion")

Behind each custom section Hybrid will add a comment, which reports the characteristics it assumes.

Hybrid will adjust the output of the script to whatever color format is configured in the encoder.

Cu Selur

Ps.: I think there is a bug somewhere, Hybrid should only set the dithering when going from a higher to lower bit depth. => uploaded a new dev which fixes that.

[andrewschen]

I'm just curious about how things work, experimenting.... 

BTW If put # requires colorformat YUV444P10 in custom script, the result shows

# adjusting output color from YUV444P10 to YUV420P10 for SvtAv1Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_in_s="limited")

If it's # requires colorformat YUV444P12:

# adjusting output color from YUV444P12 to YUV420P10 for SvtAv1Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion") 

and if it's # requires colorformat YUV444P16:

# adjusting output color from YUV444P16 to YUV420P10 for SvtAv1Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10)

why there's no dithering if it's 16bit->10bit? because it's already too good?

[Selur]

No, dithering should be there, whenever going from a higher to a lower bit depth.
Did you use the latest dev?

[Selur]

Yes, should be fixed in latest dev, at least I get:

# clip current meta; color space: YUV444P16, bit depth: 16, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive, full height: true
# adjusting output color from YUV444P16 to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion")

Cu Selur

[andrewschen]

Yes, just dl & tested, fixed in the latest dev, it now show

clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion")