Selur's Little Message Board

When trying to run ArtCNN models in VSMLRT, be it TRT, DML, or ONNX Cuda modes, it produces an error.
According to the github, "ArtCNN is supported by vs-mlrt".
It seems to be complaining about the type of data being fed to it, so possibly may need an override for planes to feed, or a workaround?

"vapoursynth.Error: operator (): expects 1 input planes"

The common models that people use are R16F96
Relevant github:
https://github.com/Artoriuz/ArtCNN
The R16F96 and R8F64 are commonly used in encoding tasks as of recent, so it'd be useful to have them available.
Specifically, the regular and chroma variants; a la ArtCNN_R8F64 and ArtCNN_R8F64_Chroma.

[attachment=3377]

Edit: Just realize the files didn't have the 1x/2x at the beginning, even when renaming the files, same plane error occurs.

Hybrid atm. uses vsmlrt.inference to call general models, for ArtCNN (like with for example RIFE, SCUNET, DPIR a separate class needs to be used) one needs to call ArtCNN:

Code:
def ArtCNN(

    clip: vs.VideoNode,

    tiles: typing.Optional[typing.Union[int, typing.Tuple[int, int]]] = None,

    tilesize: typing.Optional[typing.Union[int, typing.Tuple[int, int]]] = None,

    overlap: typing.Optional[typing.Union[int, typing.Tuple[int, int]]] = None,

    model: ArtCNNModel = ArtCNNModel.ArtCNN_C16F64,

    backend: backendT = Backend.OV_CPU()

) -> vs.VideoNode:

and only specific models are supported:

Code:
class ArtCNNModel(enum.IntEnum):

    ArtCNN_C4F32 = 0

    ArtCNN_C4F32_DS = 1

    ArtCNN_C16F64 = 2

    ArtCNN_C16F64_DS = 3

    ArtCNN_C4F32_Chroma = 4

    ArtCNN_C16F64_Chroma = 5

    ArtCNN_R16F96 = 6

    ArtCNN_R8F64 = 7

    ArtCNN_R8F64_DS = 8

    ArtCNN_R8F64_Chroma = 9

    ArtCNN_C4F16 = 10

    ArtCNN_C4F16_DS = 11

    ArtCNN_R16F96_Chroma = 12

    ArtCNN_C4F16_DN = 13

    ArtCNN_C4F32_DN = 14

    ArtCNN_R8F64_JPEG420 = 15

    ArtCNN_R8F64_JPEG444 = 16

Atm. you would need to write some custom code to use this.
ArtCNN seems to expect either GRAYH or GRAYS input,...
Something like:

Code:
# adjusting color space from YUV420P8 to GRAYH for ArtCNN

clip = core.resize.Bicubic(clip=clip, format=vs.GRAYH, matrix_in_s="470bg")

# filtering using VSMLRT

clip = vsmlrt.ArtCNN(clip,model=2, backend=Backend.TRT_RTX(fp16=True,device_id=0,num_streams=1,verbose=True,use_cuda_graph=True,workspace=1073741824,builder_optimization_level=3,engine_folder="J:/TRT"))

does seem to do something, but no clue atm. how to properly hand this. Smile

Are ArtCNN Models meant to be only applied on a specific plane of the source?
Looking at https://github.com/Artoriuz/ArtCNN it seems like different models expect different input types.

=> I'll try to look at it in the next few days (probably won't get around to it today), if you can provide some links to documentation (and/or examples) how to use these, please do.

Cu Selur

Ps.: I'll also have to update the vs-mlrt add-on since atm. it only comes with a few of the ArtCNN models.

Had a quick go at it, and I can easily implement support for it.

Code:
ArtCNN_C4F32 = 0 supports GRAYS, GRAYH

    ArtCNN_C4F32_DS =  supports GRAYS, GRAYH

    ArtCNN_C16F64 = 2 supports GRAYS, GRAYH

    ArtCNN_C16F64_DS = 3 supports GRAYS, GRAYH

    ArtCNN_C4F32_Chroma = 4 supports YUV444PS, YUV44PH

    ArtCNN_C16F64_Chroma = 5 supports YUV444PS, YUV44PH

    ArtCNN_R16F96 = 6 supports GRAYS, GRAYH

    ArtCNN_R8F64 = 7 supports GRAYS, GRAYH

    ArtCNN_R8F64_DS = 8 supports GRAYS, GRAYH

    ArtCNN_R8F64_Chroma = 9 supports YUV444PS, YUV44PH

    ArtCNN_C4F16 = 10 supports GRAYS, GRAYH

    ArtCNN_C4F16_DS = 11 supports GRAYS, GRAYH

    ArtCNN_R16F96_Chroma = 12 supports YUV444PS, YUV44PH

    ArtCNN_C4F16_DN = 13 supports GRAYS, GRAYH

    ArtCNN_C4F32_DN = 14 supports GRAYS, GRAYH

    ArtCNN_R8F64_JPEG420 = 15 supports RGBS, RGBH

    ArtCNN_R8F64_JPEG444 = 16 supports RGBS, RGBH

But I still need to know what to do with the Gray color spaces.
Should just the luma be filtered and the chroma should get copied?
Should each plane be filtered differently?

I suspect:

Code:
# Imports

import vapoursynth as vs

# getting Vapoursynth core

import logging

import site

import sys

import os

core = vs.core

# Limit frame cache to 48449MB

core.max_cache_size = 48449

# Import scripts folder

scriptPath = 'F:/Hybrid/64bit/vsscripts'

sys.path.insert(0, os.path.abspath(scriptPath))

os.environ["CUDA_MODULE_LOADING"] = "LAZY"

# Force logging to std:err

logging.StreamHandler(sys.stderr)

# loading plugins

core.std.LoadPlugin(path="F:/Hybrid/64bit/vs-mlrt/vstrt_rtx.dll")

core.std.LoadPlugin(path="F:/Hybrid/64bit/vs-mlrt/vstrt.dll")

core.std.LoadPlugin(path="F:/Hybrid/64bit/vs-mlrt/vsort.dll")

core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")

core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/BestSource/BestSource.dll")

# Import scripts

from importlib.machinery import SourceFileLoader

vsmlrt = SourceFileLoader('vsmlrt', 'F:/Hybrid/64bit/vs-mlrt/vsmlrt.py').load_module()

from vsmlrt import Backend

import validate

# Source: 'G:\TestClips&Co\files\test.avi'

# Current color space: YUV420P8, bit depth: 8, resolution: 640x352, frame rate: 25fps, scanorder: progressive, yuv luminance scale: limited, matrix: 470bg, format: MPEG-4 Visual

# Loading G:\TestClips&Co\files\test.avi using BestSource)

clip = core.bs.VideoSource(source="G:/TestClips&Co/files/test.avi", cachepath="J:/tmp/test_bestSource", track=0, hwdevice="opencl")

frame = clip.get_frame(0)

# setting color matrix to 470bg.

clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT470_BG)

# setting color transfer (vs.TRANSFER_BT601), if it is not set.

if validate.transferIsInvalid(clip):

  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT601)

# setting color primaries info (to vs.PRIMARIES_BT470_BG), if it is not set.

if validate.primariesIsInvalid(clip):

  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT470_BG)

# setting color range to TV (limited) range.

clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED)

# making sure frame rate is set to 25fps

clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)

# making sure the detected scan type is set (detected: progressive)

clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE)# progressive

# changing range from limited to full range for vsVSMLRTFilter

clip = core.resize.Bicubic(clip, format=vs.YUV420P8, range_in_s="limited", range_s="full")

# setting color range to PC (full) range.

clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_FULL)

# adjusting color space from YUV420P8 to YUV444PH for vsVSMLRTFilter

clip = core.resize.Bicubic(clip=clip, format=vs.YUV444PH)

# filtering using VSMLRT

clipY,clipU,clipV = core.std.SplitPlanes(clip)# split YUV planes to filter Y only

clip = vsmlrt.ArtCNN(clipY,model=3, backend=Backend.TRT_RTX(fp16=True,device_id=0,num_streams=1,verbose=True,use_cuda_graph=True,workspace=1073741824,builder_optimization_level=3,engine_folder="J:/TRT"))

clip = core.std.ShufflePlanes(clips=[clipY, clipU, clipV], planes=[0, 0, 0], colorfamily=vs.YUV)# merge YUV planes

# changing range from full to limited range for vsVSMLRTFilter

clip = core.resize.Bicubic(clip, format=vs.YUV444PH,range_in_s="full", range_s="limited")

# setting color range to TV (limited) range.

clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED)

# no resizing since resolution is already archived

# adjusting output color from: YUV444PH to YUV420P10 for NVEncModel

clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, dither_type="error_diffusion")

# set output frame rate to 25fps (progressive)

clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)

# output

clip.set_output()

would be a correct way to apply one of the GRAY only models, but I'm not sure. Smile

=> Can you answer this?

Cu Selur

Ps.: Based on my assumption, I uploaded a new dev and vs-mlrt add-on with ArtCNN support in VSMLRT(Filter).

GunplaXross2

Selur

Selur