Posts: 526
Hello Selur,
I just completed the development of a Vapoursynth filter for
Deoldify .
As you probably already know, Deoldify is a Deep Learning based project for colorizing and restoring old images and video.
Currently it is possible to use Deoldify using Jupyter or Stable Diffusion.
Now with this implementation it will be possible to use it directly in Vapoursynth.
To use this filter it is necessary to install fastai version v1.0.60 with the command
python -m pip fastai=1.0.60
Deoldify is delivered with a own version of fastai, so this installation is necessary to install "only" the dependencies.
After the installation of fastai it is necessary to delete it from "Hybrid\64bit\Vapoursynth\Lib\site-packages" to avoid conflicts with the Deodify version (it seems that this problem arise only when Deoldify is inside a packeage).
To install the filter it is necessary to unzip the attached file vsdeoldify-1.0.0.zip in "Hybrid\64bit\Vapoursynth\Lib\site-packages"
I don't have included the models. They must downloaded from:
https://github.com/jantic/DeOldify
And installed in the folder: "Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\models"
For your convenience I report here the links
ColorizeArtistic_gen.pth
ColorizeStable_gen.pth
ColorizeVideo_gen.pth
I attached also the file Example.zip with a simple example.
The usage is the following:
from vsdeoldify import ddeoldify
This is a draft version. It would be nice if you can install on your side to check if is working on your installation.
Thanks,
Dan
Attached Files Example.zip (Size: 222,88 KB / Downloads: 19)
vsdeoldify-1.0.0.zip (Size: 239,67 KB / Downloads: 15)
Posts: 9.913
Nice.
Posts: 526
(27.02.2024, 18:37) Selur Wrote:
It depends on the model used, the Video model in "deoldify" has been calibrated to be flicker free in videos. Unfortunately in "ddcolor" all the models have some flickering.
The colors in "ddcolor" are more saturated while "deoldify" is more conservative and the color is desatured in order to avoid the flickering.
The 2 models have different behavior and maybe the best result can be reached using them in combination (see Merge()).
(27.02.2024, 18:37) Selur Wrote:
Yes the render_factor controls the number of iteration for the convergence of the colors to be assigned to the image.
Higher values decrease the speed.
I forget to specify that a reasonable range of values for render_factor is between 10 and 44, being 21 a good default.
In term of speed render_factor affect the speed as input_size in "ddcolor".
With render_factor=30, "deoldify" has about the same speed of "ddcolor" with input_size =1024.
Posts: 9.913
27.02.2024, 20:19
(This post was last modified: 27.02.2024, 20:53 by Selur .)
Unexpectedly I had a few minutes time and I did the following:
cd f:\Hybrid\64bit\Vapoursynth\
which showed:
Collecting fastai==1.0.60
so far so good.
Then I called:
python -m pip uninstall fastai==1.0.6
which should onyl remove fastai, but keep the dependencies.
Found existing installation: fastai 1.0.60
That worked as expected.
I then extracted the contents of vsdeoldify-1.0.0.zip to
f:\hybrid\64bit\vapoursynth\lib\site-packages
and the models to:
f:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\models
(Okay, I see why you didn't include those models.)
To test it, I did the following.
start Hybrid
load a source
enabled 'Filtering->Vapoursynth->Color->Basic->Tweak' and set 'Saturation' to 0.
I then set 'Filtering->Vapoursynth->Custim->Insert before' to 'Grayworld', enabled the edit section and for quick test entered:clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
Downloading: "https://download.pytorch.org/models/resnet101-63fe2227.pth" to C:\Users\Selur/.cache\torch\hub\checkpoints\resnet101-63fe2227.pth
Posts: 9.913
Switchting to model 2 also triggers a download:
Downloading: "https://download.pytorch.org/models/resnet34-b627a593.pth" to C:\Users\Selur/.cache\torch\hub\checkpoints\resnet34-b627a593.pth
for amusement:
https://imgsli.com/MjQyOTky
side note, next release will again support properly handling the custom hinting:
# requires colorformat RGB24
the '# requires colorformat RGB24' lets Hybrid know that RGB24 is required.
Main issue atm. I see is the additional downloads, into a folder which probably never gets cleaned up.
Cu Selur
Posts: 526
I'm happy to know that is working as expected.
The resnet network is necessary for torch and is automatically downloaded (only the first time).
It is the backbone of the generator- objects are detected more consistently and correctly with this.
I cannot do nothing on that issue.
As you can note the colors provided by ddcolor are more saturated of the colors applied by deoldify.
For amusement I compared the effect of render_factor and input_size on deoldify and ddcolor
https://imgsli.com/MjQzMDM1
The main differences are in the color of the hair and hands.
As I wrote none of the filter is perfect, maybe it is possible to obtain a better result by combining them.
Are you planning to add this filter in Hybrid ?
Thanks,
Dan
Posts: 9.913
28.02.2024, 05:51
(This post was last modified: 28.02.2024, 06:20 by Selur .)
Quote: The main differences are in the color of the hair and hands.
It's all the colors. See the color of the tree (on the right) and the floors.
Quote: Are you planning to add this filter in Hybrid ?
Not sure atm., will have to test whether:
a. one can download the "resnet101-63fe2227.pth" models beforehand and maybe place them in the models folder (<- that doesn't work).
I really don't like the idea of Hybrid triggering 'random' downloads.
b. I manage to put alle the dependencies&co together so that there is a single download I could add as a spearate addon-package. (alternatively writing down the steps to install this manually would also be possible, but I prefer the package, since it better allows to reproduce problems)
Quote: As I wrote none of the filter is perfect, maybe it is possible to obtain a better result by combining them.
Not sure whether that will work, since iirc. both of the filters assume gray colored content and don't try to just fix some colors.
I doubt, something like a simple merging or some masked merges will work.
Also the video model does seem to add some blue halos.
Cu Selur
Ps.: main issue for me are the additional downloads. Not sure whether changing 'TORCH_HOME' environment variable a. will work b. will not cause issues with other filters.
Creating 'f:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\models\resnet\hub\checkpoints' with the files in it and using:
import os
does seem to work, but no clue whether it causes any issues with other filters (mlrt or torch filters).
Posts: 526
The download of resnet101 is triggered by "fastai vision" in the files: presnet.py, xresnet.py, xresnet2.py
The function used is defined in torch.utils.model_zoo.load_url()
The logic is embedded in fastai version of Deoldify. More precisely in the function
def xresnet101(pretrained=False, **kwargs):
While the use of the ".cache" is embedd directly in torch, I don't think that is possible to change this logic.
You can find more info here:
https://pytorch.org/docs/stable/hub.html
In meanwhile I discovered that is not a problem to have "fastai==1.0.60" installed with vsdeoldify.
It was a problem for me during the development, but once all the "imports" are correctly assigned, the problem disappears.
I fixed some small issue in the package, I attached a new version.
Thanks,
Dan
Attached Files vsdeoldify-1.0.2.zip (Size: 239,57 KB / Downloads: 9)
Posts: 526
I changed the file __init__.py adding the following code
torch.hub.set_dir(model_dir)
now the "resnet" files are saved in the directory: "Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\models\checkpoints"
I attached the new version of the filter.
Dan
Attached Files vsdeoldify-1.0.3.zip (Size: 239,74 KB / Downloads: 13)
Posts: 9.913
So if they are downloaded beforehand, there should be no need to download them, right?
![[Image: grafik.png]](https://i.ibb.co/P91QLsc/grafik.png)
![[Image: grafik.png]](https://i.ibb.co/Wx2961Z/grafik.png)
)![[Image: grafik.png]](https://i.ibb.co/P5QR9pQ/grafik.png)
![[Image: grafik.png]](https://i.ibb.co/5FgCxNv/grafik.png)
![[Image: grafik.png]](https://i.ibb.co/87zhpvQ/grafik.png)