Deoldify Vapoursynth filter

[Selur]

Sounds good.
Have to taken to handle scene changes properly?

Cu Selur

[zspeciman]

The Preview doesn't give any errors, just a blank page, see attached picture.  Pressing play is still blank.   It works fine and gives pictures when DeOldify or DDColor are separately by themselves.  

Looking forward to this "3.0.0 I will delay the import of vsddcolor", thank you.  Maybe that will solve it.

I can see that a GPU upgrade will be necessary at some point in the near future. 
What GPUs do either  one of you use or suggest?
Is it possible to run two GPUs on Hybrid, or I'll just have to retire the 1080ti?

[Selur]

What GPUs do either one of you use or suggest?

fastest in your budget (using a RTX 4080 myself)

What GPUs do either one of you use or suggest?

You can't use 2 gpus for the same filter, but you can use GPU1 in filter A and GPU2 in filter B, but I doubt that will really help much.

What happens if you:
a. copy the content of the script preview
b. open vsViewer
c. insert the content there and press F5

Cu Selur

[zspeciman]

NICE, the RTX 4080, thats the Rolls Royce of these cards

"What happens if you:
a. copy the content of the script preview
b. open vsViewer
c. insert the content there and press F5"

it doesn't do anything. Still blank preview.
The compare view that normally gives side by side before and after, just gives the same blank preview.

[Selur]

You should see some error message under the text field where you entered the copied script.

[zspeciman]

Got it, here it is:

2024-03-28 06:16:20.690
Qt warning: QPixmap::scaled: Pixmap is a null pixmap
2024-03-28 06:16:21.339
Error on frame 0 request:

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 2941, in vapoursynth.publicFunction
File "src\cython\vapoursynth.pyx", line 2943, in vapoursynth.publicFunction
File "src\cython\vapoursynth.pyx", line 683, in vapoursynth.FuncData.__call__
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\__init__.py", line 94, in inference
output_ab = module(img_gray_rgb)
^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch.py", line 59, in forward
out_feat = self.decoder()
^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch.py", line 113, in forward
out = self.color_decoder([out0, out1, out2], out3)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch.py", line 305, in forward
output = self.transformer_cross_attention_layers[i](
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch_utils\transformer_utils.py", line 124, in forward
return self.forward_post(tgt, memory, memory_mask,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch_utils\transformer_utils.py", line 93, in forward_post
tgt2 = self.multihead_attn(query=self.with_pos_embed(tgt, query_pos),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\activation.py", line 1241, in forward
attn_output, attn_output_weights = F.multi_head_attention_forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\functional.py", line 5440, in multi_head_attention_forward
attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: cutlassF: no kernel found to launch!

[Selur]

Since DDColor does work on it's own.
Does it work when:
a. FP16 is enabled
b. FP16 is disabled
?

Also, try whether DeOldify with DDColor Combine 'Simple', works if you use lower render factor (i.e. 10) in both DeOldify and DDColor Settings.

Cu Selur

[zspeciman]

DDColor does not work with FP16 enabled, here is the vsviewer error

2024-03-28 06:38:46.829
Qt warning: QPixmap:caled: Pixmap is a null pixmap
2024-03-28 06:38:47.533
Error on frame 0 request:

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 2941, in vapoursynth.publicFunction
File "src\cython\vapoursynth.pyx", line 2943, in vapoursynth.publicFunction
File "src\cython\vapoursynth.pyx", line 683, in vapoursynth.FuncData.__call__
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\__init__.py", line 94, in inference
output_ab = module(img_gray_rgb)
^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch.py", line 59, in forward
out_feat = self.decoder()
^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch.py", line 113, in forward
out = self.color_decoder([out0, out1, out2], out3)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch.py", line 305, in forward
output = self.transformer_cross_attention_layers[i](
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch_utils\transformer_utils.py", line 124, in forward
return self.forward_post(tgt, memory, memory_mask,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch_utils\transformer_utils.py", line 93, in forward_post
tgt2 = self.multihead_attn(query=self.with_pos_embed(tgt, query_pos),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\activation.py", line 1241, in forward
attn_output, attn_output_weights = F.multi_head_attention_forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\functional.py", line 5440, in multi_head_attention_forward
attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: cutlassF: no kernel found to launch!


With Render factor at 10 and Combine Simple  also doesn't work, here is the vsviewer error
2024-03-28 06:45:51.129
Qt warning: QPixmap:caled: Pixmap is a null pixmap
2024-03-28 06:45:51.715
Error on frame 0 request:

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 2941, in vapoursynth.publicFunction
File "src\cython\vapoursynth.pyx", line 2943, in vapoursynth.publicFunction
File "src\cython\vapoursynth.pyx", line 683, in vapoursynth.FuncData.__call__
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\__init__.py", line 94, in inference
output_ab = module(img_gray_rgb)
^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch.py", line 59, in forward
out_feat = self.decoder()
^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch.py", line 113, in forward
out = self.color_decoder([out0, out1, out2], out3)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch.py", line 305, in forward
output = self.transformer_cross_attention_layers[i](
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch_utils\transformer_utils.py", line 124, in forward
return self.forward_post(tgt, memory, memory_mask,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsddcolor\ddcolor_arch_utils\transformer_utils.py", line 93, in forward_post
tgt2 = self.multihead_attn(query=self.with_pos_embed(tgt, query_pos),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\modules\activation.py", line 1241, in forward
attn_output, attn_output_weights = F.multi_head_attention_forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\nn\functional.py", line 5440, in multi_head_attention_forward
attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: cutlassF: no kernel found to launch!

[Selur]

That's what I suspected, seems like older cards do not support FP16.
Try opening Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\__init__.py inside a text editor and change

# adjusting clip's color space to RGBH for vsDDColor
    clipb = ddcolor(clipb.resize.Bicubic(format=vs.RGBH, range_s="limited"), model=model, input_size=input_size, device_index=device_index, num_streams=num_streams)

to

# adjusting clip's color space to RGBS for vsDDColor
    clipb = ddcolor(clipb.resize.Bicubic(format=vs.RGBS, range_s="limited"), model=model, input_size=input_size, device_index=device_index, num_streams=num_streams)

this probably should 'fix' the problem.
@Dan64: Maybe add an FP16 option for DDColor, depending on which DDColor is called with RGBS or RGBH content?

Cu Selur

[zspeciman]

Its Working now   Bravo!!!

How much faster is having the FP16 enabled?