Deoldify Vapoursynth filter

[zspeciman]

Thank you for the all the updates.
On the latest Hbyrid rev 2024.03.26.1, VapoursynthR65_torch_2024.03.24, vs-mlrt_2024.03.26

DDColor enabled only, with defaults Artistic, Input size 512.  All works including preview

DeOldify enabled only, with model Video; DDColor Combine = DeOldify Only.  All works including preview

The crash happens when DDcolor Combine with all other models.  The preview doesn't work either. 

I've attached the Debug file for the DDColor Combine=Simple
Also a separate Debug for the Preview, in case that helps find the problem.

[Dan64]

Sorry, but if you want to use Deoldify and DDcolor together it is better to use directly DDeoldify that support the combined use.
So disable DDcolor and use only DDeoldify with "DDColor Combine = Simple" and set "Merge weight = 0"
Enable also "Resize chroma".

Dan

P.S.
What you are doing do not make sense, you are using DDColor to color the clip and then you provide the colored clip in input to DDeoldify, but this not makes sense, because all the color information is ignored by the color filters.
You are only losing time.

[zspeciman]

Thanks Dan, that makes sense.  
I followed your instructions, but I still get the crash under your suggested settings. 
There is something about the combining the two that gives the crash.  Each on their own work fine
I've attached the Debug of the crash and of the preview

[Dan64]

For this case of problems only the preview log is useful, because vspipe is not able to propagate the error messages provided by the filters.

For what I see the error is raised by vsddcolor in the function "inference".

I cannot reproduce the problem.

Try to reinstall vs-ddcolor.

Otherwise, install last Hybrid version and download the last torch package provided by Selur on the download page.

Dan

[Selur]

Looking at the ddoldify-call:

clip = ddeoldify(clip=clip, model=1, sat=[1.00,1.00], hue=[0.00,0.00], chroma_resize=True, dd_method=2, merge_weight=0.00, dd_method_params=[0.6,2,0.15,0.2,False], dd_tweak_luma_bind=[False,0.4,0.4], dd_model=1, dd_tweak_bright=0.00)

"merge_weight=0.00" is rather odd, since setting it to 0 does disable the ddColor here, but the call does work fine here.

Reset your Deoldify defaults.
Are your gpu drivers up-to-date?
What gpu are you using?
Does it work with lower resolution inputs?

Cu Selur

Ps.: using enabling DDColor + DeOldify does make no sense, since both will reset the content to grayscale, so whatever gets applied first will effectivly do nothing aside from eating up ressources. (Like Dan64 wrote, use DeOldifys controls)

[Dan64]

So disable DDcolor and use only DDeoldify with "DDColor Combine = Simple" and set "Merge weight = 0"

I proposed " Merge weight = 0 " because was the setting used by you. But I agree with Selur, if you want to use the combined model it is better assign a weight to the merge.
Good values are in a range [0.2-0.5]. I suggest to use 0.4, because DDColor used for coloring movies it too unstable and it is better to assign more weight to Deoldify.

Dan

[zspeciman]

I've reinstalled Hybrid, Vapoursynth latest version, everything is reset to defaults.  Enabled Deoldify and DDColor Combine = Simple, weight 0.5, still doesn't work, preview doesn't work either; it pop ups, but the screen has no image. 
The interesting thing is, if I Enable DDColor by itself, it works. If I enable DeOldify by itself with DDColor Combine=DDColor Only, and everything else unchecked it doesn't work.  Aren't these two equivalent?
My GPU is  NVIDIA GeForce GTX 1080 Ti and it is on the latest driver.  The cpu is AMD Ryzen 5 5600X
On the previous Hybrid, I had the same issue, when combining the two.  
The clip I tested was the Dan has uploaded, that of the Chapel.  I tried other clips, same issue. 


I reinstalled the very 1st Hybrid 024.03.03.1 / VapoursynthR65_torch_2024.03.03 where the coloring was introduced, just to test the DDcolor Combine. 
Same issue, DDColor by itself works.  DeOldify DDColor Combine = no DDColor works.  DeOldify by itself DDColor Combine = Simple Merge, Weight 0.50, it crashes, no preview
Maybe the Debug and Preview report on that one can help, I've attached those

[Selur]

old version -> no support from me.

Deinstall Hybrid with all settings.
Install the latest downloads.
Try the preview.
Share the error messages shown in the vsViewer Window.

Cu Selur

Ps.: As a side note: vs-mlrt just switched to trt 10.0.0 + cuda 12.4.0 + cudnn 8.9.7, which requires a minimum driver version of 525 and is compatible with 16/20 series and newer GPUs. The engine compilation time is reduced by up to 40%, but the runtime performance of RIFE models is worsen by up to 30% with nearly doubled => starting with the next vs-mlrt release, the 10XX-cards are no longer supported with vs-mlrt.

[Dan64]

The interesting thing is, if I Enable DDColor by itself, it works. If I enable DeOldify by itself with DDColor Combine=DDColor Only, and everything else unchecked it doesn't work.  Aren't these

In the log that you provided there is this line

# adding colors using DeOldify
from vsdeoldify import ddeoldify
clip = ddeoldify(clip=clip, model=0, sat=[1.00,1.00], hue=[0.00,0.00], dd_method=0, dd_weight=0.50, dd_model=1)

 
the dd_method=0 activate only Deoldify, so probably you selected " DDColor Combine=DeOldify Only"

But the error is reported on __init__.py of DDColor.

The only line in the filter that can activate this function is the following

os.environ["CUDA_MODULE_LOADING"] = "LAZY"
os.environ["NUMEXPR_MAX_THREADS"] = "8"

from vsddcolor import ddcolor

In the file __init__.py of DDColor there are the following lines

from __future__ import annotations

import os
from threading import Lock

import kornia
import numpy as np
import torch
import torch.nn.functional as F
import vapoursynth as vs

from .ddcolor_arch import DDColor

__version__ = "1.0.0"

os.environ["CUDA_MODULE_LOADING"] = "LAZY"

model_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), "models")

 
the failure is reported in the torch import.
But if this is the case, also DDColor stand-alone should not work.

I'm sorry but I can't reproduce this problem.

Probably an upgrade of GPU to RTX 20/30 could help.

Dan

---

In the next release 3.0.0 I will delay the import of vsddcolor, so that it will be loaded only if really necessary.
Maybe in this way you will be able to use DeOldify.

Dan

[Dan64]

Hello Selur,

  I found a way to mitigate the problem of gray frames in AverageFrames.

  As you know I developed a filter which try to stabilize the colors of a clip using color temporal stabilizer.
  
  The stabilization is performed by averaging past/future frames.
  Since the non matched areas of past/future frames are gray because is missing in the past/future the color information, the new version of the filter will try to mitigate this issue, by filling the gray areas with the pixels of current frames (eventually de-saturated with the parameter "sat"). 
  The image restored in this way is blended with the non restored image using the parameter "weight" (if =0 no blending is applied).
   The gray areas are selected by the threshold parameter "tht". All the pixels in the HSV color space with "S" < "tht" will be considered gray. If "tht=0" no color frame restore is applied.

   Here an example of asking to Vapoursynth to provide the image of the past 2 frames.

  

   the gray areas are the pixel not available in the current frame, which is the following

  

   It is possible identify the gray areas in the HSV color space, and selecting the pixels with the plane "S" below a given threshold.

   Here an example of mask with threshold = 50:

  

   And here with threshold = 100

   

    I think that a threshold = 80, could be a good compromise.

    He the gray image restored with this method, using a threshold=80 and a blending weight=50%

     

     Using a blending=0 it will be possible to obtain a more colored picture but in this way it will be lost the stabilization effect of the Average.

Dan