ArtCNN issue/feature request

[Selur]

Had a quick go at it, and I can easily implement support for it.

ArtCNN_C4F32 = 0 supports GRAYS, GRAYH ArtCNN_C4F32_DS = supports GRAYS, GRAYH ArtCNN_C16F64 = 2 supports GRAYS, GRAYH ArtCNN_C16F64_DS = 3 supports GRAYS, GRAYH ArtCNN_C4F32_Chroma = 4 supports YUV444PS, YUV44PH ArtCNN_C16F64_Chroma = 5 supports YUV444PS, YUV44PH ArtCNN_R16F96 = 6 supports GRAYS, GRAYH ArtCNN_R8F64 = 7 supports GRAYS, GRAYH ArtCNN_R8F64_DS = 8 supports GRAYS, GRAYH ArtCNN_R8F64_Chroma = 9 supports YUV444PS, YUV44PH ArtCNN_C4F16 = 10 supports GRAYS, GRAYH ArtCNN_C4F16_DS = 11 supports GRAYS, GRAYH ArtCNN_R16F96_Chroma = 12 supports YUV444PS, YUV44PH ArtCNN_C4F16_DN = 13 supports GRAYS, GRAYH ArtCNN_C4F32_DN = 14 supports GRAYS, GRAYH ArtCNN_R8F64_JPEG420 = 15 supports RGBS, RGBH ArtCNN_R8F64_JPEG444 = 16 supports RGBS, RGBH

But I still need to know what to do with the Gray color spaces.
Should just the luma be filtered and the chroma should get copied?
Should each plane be filtered differently?

I suspect:

# Imports import vapoursynth as vs # getting Vapoursynth core import logging import site import sys import os core = vs.core # Limit frame cache to 48449MB core.max_cache_size = 48449 # Import scripts folder scriptPath = 'F:/Hybrid/64bit/vsscripts' sys.path.insert(0, os.path.abspath(scriptPath)) os.environ["CUDA_MODULE_LOADING"] = "LAZY" # Force logging to std:err logging.StreamHandler(sys.stderr) # loading plugins core.std.LoadPlugin(path="F:/Hybrid/64bit/vs-mlrt/vstrt_rtx.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vs-mlrt/vstrt.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vs-mlrt/vsort.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/fmtconv.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/BestSource/BestSource.dll") # Import scripts from importlib.machinery import SourceFileLoader vsmlrt = SourceFileLoader('vsmlrt', 'F:/Hybrid/64bit/vs-mlrt/vsmlrt.py').load_module() from vsmlrt import Backend import validate # Source: 'G:\TestClips&Co\files\test.avi' # Current color space: YUV420P8, bit depth: 8, resolution: 640x352, frame rate: 25fps, scanorder: progressive, yuv luminance scale: limited, matrix: 470bg, format: MPEG-4 Visual # Loading G:\TestClips&Co\files\test.avi using BestSource) clip = core.bs.VideoSource(source="G:/TestClips&Co/files/test.avi", cachepath="J:/tmp/test_bestSource", track=0, hwdevice="opencl") frame = clip.get_frame(0) # setting color matrix to 470bg. clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT470_BG) # setting color transfer (vs.TRANSFER_BT601), if it is not set. if validate.transferIsInvalid(clip): clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT601) # setting color primaries info (to vs.PRIMARIES_BT470_BG), if it is not set. if validate.primariesIsInvalid(clip): clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT470_BG) # setting color range to TV (limited) range. clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED) # making sure frame rate is set to 25fps clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1) # making sure the detected scan type is set (detected: progressive) clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE)# progressive # changing range from limited to full range for vsVSMLRTFilter clip = core.resize.Bicubic(clip, format=vs.YUV420P8, range_in_s="limited", range_s="full") # setting color range to PC (full) range. clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_FULL) # adjusting color space from YUV420P8 to YUV444PH for vsVSMLRTFilter clip = core.resize.Bicubic(clip=clip, format=vs.YUV444PH) # filtering using VSMLRT clipY,clipU,clipV = core.std.SplitPlanes(clip)# split YUV planes to filter Y only clip = vsmlrt.ArtCNN(clipY,model=3, backend=Backend.TRT_RTX(fp16=True,device_id=0,num_streams=1,verbose=True,use_cuda_graph=True,workspace=1073741824,builder_optimization_level=3,engine_folder="J:/TRT")) clip = core.std.ShufflePlanes(clips=[clipY, clipU, clipV], planes=[0, 0, 0], colorfamily=vs.YUV)# merge YUV planes # changing range from full to limited range for vsVSMLRTFilter clip = core.resize.Bicubic(clip, format=vs.YUV444PH,range_in_s="full", range_s="limited") # setting color range to TV (limited) range. clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED) # no resizing since resolution is already archived # adjusting output color from: YUV444PH to YUV420P10 for NVEncModel clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion") # set output frame rate to 25fps (progressive) clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1) # output clip.set_output()

would be a correct way to apply one of the GRAY only models, but I'm not sure.

=> Can you answer this?

Cu Selur

Ps.: Based on my assumption, I uploaded a new dev and vs-mlrt add-on with ArtCNN support in VSMLRT(Filter).