Selur's Little Message Board

Hello Selur,

   I just completed the development of a Vapoursynth filter for Deoldify.
   As you probably already know, Deoldify is a Deep Learning based project for colorizing and restoring old images and video.
  
   Currently it is possible to use Deoldify using Jupyter or Stable Diffusion.

   Now with this implementation it will be possible to use it directly in Vapoursynth.

   To use this filter it is necessary to install fastai version v1.0.60 with the command


  Deoldify is delivered with a own version of fastai, so this installation is necessary to install "only" the dependencies.
  After the installation of fastai it is necessary to delete it from "Hybrid\64bit\Vapoursynth\Lib\site-packages" to avoid conflicts with the Deodify version (it seems that this problem arise only when Deoldify is inside a packeage).
 
  To install the filter it is necessary to unzip the attached file vsdeoldify-1.0.0.zip in "Hybrid\64bit\Vapoursynth\Lib\site-packages"

  I don't have included the models. They must downloaded from: https://github.com/jantic/DeOldify
  And installed in the folder:  "Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\models"  

  For your convenience I report here the links

   ColorizeArtistic_gen.pth
   ColorizeStable_gen.pth
   ColorizeVideo_gen.pth

   I attached also the file Example.zip with a simple example.

  The usage is the following:


  This is a draft version. It would be nice if you can install on your side to check if is working on your installation.

Thanks,
Dan

Nice.
Sadly, I'm busy today and tomorrow, but I'll test it and report back on Thursday after work.
How does it compare to ddcolor?

iirc render_factor slowed down deolidfy quite a bit, but helped with stabilizing the colors. (iirc. one needed to use 20+ to get usable results)
do you have any experience with this (it's quite a while since I last used deoldify)

Cu Selur

(27.02.2024, 18:37)Selur Wrote: [ -> ]How does it compare to ddcolor?

  It depends on the model used, the Video model in "deoldify" has been calibrated to be flicker free in videos. Unfortunately in "ddcolor" all the models have some flickering.   
  The colors in "ddcolor" are more saturated while "deoldify" is more conservative and the color is desatured in order to avoid the flickering.
  The 2 models have different behavior and maybe the best result can be reached using them in combination (see Merge()).

---

(27.02.2024, 18:37)Selur Wrote: [ -> ]iirc render_factor slowed down deolidfy quite a bit, but helped with stabilizing the colors. (iirc. one needed to use 20+ to get usable results)
do you have any experience with this (it's quite a while since I last used deoldify)

Yes the render_factor controls the number of iteration for the convergence of the colors to be assigned to the image.
Higher values decrease the speed.
I forget to specify that a reasonable range of values for render_factor is between 10 and 44, being 21 a good default.
In term of speed render_factor affect the speed as input_size in "ddcolor".    
With render_factor=30, "deoldify" has about the same speed of "ddcolor" with  input_size =1024.

Unexpectedly I had a few minutes time and I did the following:
which showed:
so far so good.
Then I called:
which should onyl remove fastai, but keep the dependencies.
That worked as expected.
I then extracted the contents of vsdeoldify-1.0.0.zip to
and the models to:
(Okay, I see why you didn't include those models.)

To test it, I did the following.

• start Hybrid
  
• load a source
  
• enabled 'Filtering->Vapoursynth->Color->Basic->Tweak' and set 'Saturation' to 0.
  
• I then set 'Filtering->Vapoursynth->Custim->Insert before' to 'Grayworld', enabled the edit section and for quick test entered:
  
  Code:

  clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
from vsdeoldify import ddeoldify
clip = ddeoldify(clip, model=0, render_factor=21)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
  when pressing the preview, this triggered, my firewall to popup,...
  Allowing the download gave:
  
  Code:

  Downloading: "https://download.pytorch.org/models/resnet101-63fe2227.pth" to C:\Users\Selur/.cache\torch\hub\checkpoints\resnet101-63fe2227.pth
Downloading: "https://download.pytorch.org/models/resnet101-63fe2227.pth" to C:\Users\Selur/.cache\torch\hub\checkpoints\resnet101-63fe2227.pth
0%|          | 0.00/171M [00:00
1%|          | 1.20M/171M [00:00
1%|▏         | 2.48M/171M [00:00
2%|▏         | 3.73M/171M [00:00
3%|▎         | 5.00M/171M [00:00
4%|▎         | 6.25M/171M [00:00
4%|▍         | 7.52M/171M [00:00
5%|▌         | 8.78M/171M [00:00
6%|▌         | 10.0M/171M [00:00
7%|▋         | 11.3M/171M [00:00
7%|▋         | 12.6M/171M [00:01
8%|▊         | 13.8M/171M [00:01
9%|▉         | 15.1M/171M [00:01
10%|▉         | 16.3M/171M [00:01
10%|█         | 17.6M/171M [00:01
11%|█         | 18.9M/171M [00:01
12%|█▏        | 20.1M/171M [00:01
13%|█▎        | 21.4M/171M [00:01
13%|█▎        | 22.7M/171M [00:01
14%|█▍        | 23.9M/171M [00:01
15%|█▍        | 25.2M/171M [00:02
16%|█▌        | 26.5M/171M [00:02
16%|█▋        | 27.7M/171M [00:02
17%|█▋        | 29.0M/171M [00:02
18%|█▊        | 30.3M/171M [00:02
18%|█▊        | 31.5M/171M [00:02
19%|█▉        | 32.8M/171M [00:02
20%|█▉        | 34.1M/171M [00:02
21%|██        | 35.4M/171M [00:02
21%|██▏       | 36.6M/171M [00:02
22%|██▏       | 37.9M/171M [00:03
23%|██▎       | 39.1M/171M [00:03
24%|██▎       | 40.4M/171M [00:03
24%|██▍       | 41.7M/171M [00:03
25%|██▌       | 42.9M/171M [00:03
26%|██▌       | 44.2M/171M [00:03
27%|██▋       | 45.5M/171M [00:03
27%|██▋       | 46.7M/171M [00:03
28%|██▊       | 48.0M/171M [00:03
29%|██▉       | 49.3M/171M [00:03
30%|██▉       | 50.5M/171M [00:04
30%|███       | 51.8M/171M [00:04
31%|███       | 53.1M/171M [00:04
32%|███▏      | 54.3M/171M [00:04
33%|███▎      | 55.6M/171M [00:04
33%|███▎      | 56.9M/171M [00:04
34%|███▍      | 58.1M/171M [00:04
35%|███▍      | 59.4M/171M [00:04
36%|███▌      | 60.7M/171M [00:04
36%|███▋      | 61.9M/171M [00:04
37%|███▋      | 63.2M/171M [00:05
38%|███▊      | 64.4M/171M [00:05
39%|███▊      | 65.7M/171M [00:05
39%|███▉      | 67.0M/171M [00:05
40%|████      | 68.2M/171M [00:05
41%|████      | 69.5M/171M [00:05
42%|████▏     | 70.8M/171M [00:05
42%|████▏     | 72.0M/171M [00:05
43%|████▎     | 73.3M/171M [00:05
44%|████▎     | 74.6M/171M [00:05
44%|████▍     | 75.8M/171M [00:06
45%|████▌     | 77.1M/171M [00:06
46%|████▌     | 78.4M/171M [00:06
47%|████▋     | 79.6M/171M [00:06
47%|████▋     | 80.9M/171M [00:06
48%|████▊     | 82.2M/171M [00:06
49%|████▉     | 83.4M/171M [00:06
50%|████▉     | 84.7M/171M [00:06
50%|█████     | 86.0M/171M [00:06
51%|█████     | 87.2M/171M [00:06
52%|█████▏    | 88.5M/171M [00:07
53%|█████▎    | 89.8M/171M [00:07
53%|█████▎    | 91.0M/171M [00:07
54%|█████▍    | 92.3M/171M [00:07
55%|█████▍    | 93.6M/171M [00:07
56%|█████▌    | 94.8M/171M [00:07
56%|█████▋    | 96.1M/171M [00:07
57%|█████▋    | 97.4M/171M [00:07
58%|█████▊    | 98.6M/171M [00:07
59%|█████▊    | 99.9M/171M [00:07
59%|█████▉    | 101M/171M [00:08
60%|██████    | 102M/171M [00:08
61%|██████    | 104M/171M [00:08
62%|██████▏   | 105M/171M [00:08
62%|██████▏   | 106M/171M [00:08
63%|██████▎   | 107M/171M [00:08
64%|██████▍   | 109M/171M [00:08
65%|██████▍   | 110M/171M [00:08
65%|██████▌   | 111M/171M [00:08
66%|██████▌   | 113M/171M [00:08
67%|██████▋   | 114M/171M [00:09
67%|██████▋   | 115M/171M [00:09
68%|██████▊   | 116M/171M [00:09
69%|██████▉   | 118M/171M [00:09
70%|██████▉   | 119M/171M [00:09
70%|███████   | 120M/171M [00:09
71%|███████   | 121M/171M [00:09
72%|███████▏  | 123M/171M [00:09
73%|███████▎  | 124M/171M [00:09
73%|███████▎  | 125M/171M [00:09
74%|███████▍  | 126M/171M [00:10
75%|███████▍  | 128M/171M [00:10
76%|███████▌  | 129M/171M [00:10
76%|███████▋  | 130M/171M [00:10
77%|███████▋  | 132M/171M [00:10
78%|███████▊  | 133M/171M [00:10
79%|███████▊  | 134M/171M [00:10
79%|███████▉  | 135M/171M [00:10
80%|████████  | 137M/171M [00:11
81%|████████  | 138M/171M [00:11
82%|████████▏ | 139M/171M [00:11
82%|████████▏ | 140M/171M [00:11
83%|████████▎ | 142M/171M [00:11
84%|████████▍ | 143M/171M [00:11
85%|████████▍ | 144M/171M [00:11
85%|████████▌ | 145M/171M [00:11
86%|████████▌ | 147M/171M [00:11
87%|████████▋ | 148M/171M [00:11
88%|████████▊ | 149M/171M [00:12
88%|████████▊ | 150M/171M [00:12
89%|████████▉ | 152M/171M [00:12
90%|████████▉ | 153M/171M [00:12
90%|█████████ | 154M/171M [00:12
91%|█████████ | 156M/171M [00:12
92%|█████████▏| 157M/171M [00:12
93%|█████████▎| 158M/171M [00:12
93%|█████████▎| 159M/171M [00:12
94%|█████████▍| 161M/171M [00:12
95%|█████████▍| 162M/171M [00:13
96%|█████████▌| 163M/171M [00:13
96%|█████████▋| 164M/171M [00:13
97%|█████████▋| 166M/171M [00:13
98%|█████████▊| 167M/171M [00:13
99%|█████████▊| 168M/171M [00:13
99%|█████████▉| 169M/171M [00:13
100%|██████████| 171M/171M [00:13
  
  => seems to work (~5.4GB VRAM used)
  Increasing the render_factor to 40 roughly doubled the VRAM usage.
  Using a 4k source didn't really increase the VRAM usage, a lot. (maybe 20-25%)
  
  (yes, deoldify isn't good with nature. )
  I then threw some 1080p content at it
  
  (again, not suited for natur content)
  
  on the last I also used ddColor for comparison:
  
  
  So conclusion: It's slow, but working.
  
  Also found a bug in my code regarding handling of custom sections.
  
  Cu Selur

Switchting to model 2 also triggers a download:
for amusement: https://imgsli.com/MjQyOTky

side note, next release will again support properly handling the custom hinting:
the '# requires colorformat RGB24' lets Hybrid know that RGB24 is required.

Main issue atm. I see is the additional downloads, into a folder which probably never gets cleaned up.

Cu Selur

I'm happy to know that is working as expected.

The resnet network is necessary for torch and is automatically downloaded (only the first time).
It is the backbone of the generator- objects are detected more consistently and correctly with this.
I cannot do nothing on that issue. 

As you can note the colors provided by ddcolor are more saturated of the colors applied by deoldify.

For amusement I compared the effect of render_factor and input_size on deoldify and ddcolor

https://imgsli.com/MjQzMDM1

The main differences are in the color of the hair and hands.

As I wrote none of the filter is perfect, maybe it is possible to obtain a better result by combining them.

Are you planning to add this filter in Hybrid ?

Thanks,
Dan

Quote:The main differences are in the color of the hair and hands.

It's all the colors. See the color of the tree (on the right) and the floors.

Quote:Are you planning to add this filter in Hybrid ?

Not sure atm., will have to test whether:
a. one can download the "resnet101-63fe2227.pth" models beforehand and maybe place them in the models folder (<- that doesn't work).
I really don't like the idea of Hybrid triggering 'random' downloads.
b. I manage to put alle the dependencies&co together so that there is a single download I could add as a spearate addon-package. (alternatively writing down the steps to install this manually would also be possible, but I prefer the package, since it better allows to reproduce problems)

Quote:As I wrote none of the filter is perfect, maybe it is possible to obtain a better result by combining them.

Not sure whether that will work, since iirc. both of the filters assume gray colored content and don't try to just fix some colors.
I doubt, something like a simple merging or some masked merges will work.

Also the video model does seem to add some blue halos.

Cu Selur

Ps.: main issue for me are the additional downloads. Not sure whether changing 'TORCH_HOME' environment variable a. will work b. will not cause issues with other filters.
Creating 'f:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\models\resnet\hub\checkpoints' with the files in it and using:
does seem to work, but no clue whether it causes any issues with other filters (mlrt or torch filters).

The download of resnet101 is triggered by "fastai vision" in the files: presnet.py, xresnet.py, xresnet2.py
The function used is defined in torch.utils.model_zoo.load_url()
The logic is embedded in fastai version of Deoldify. More precisely in the function  

While the use of the ".cache" is embedd directly in torch, I don't think that is possible to change this logic.

You can find more info here: https://pytorch.org/docs/stable/hub.html

In meanwhile I discovered that is not a problem to have "fastai==1.0.60" installed with vsdeoldify.
It was a problem for me during the development, but once all the "imports" are correctly assigned, the problem disappears.

I fixed some small issue in the package, I attached a new version.

Thanks,
Dan

I changed the file __init__.py adding the following code


now the "resnet" files are saved in the directory: "Hybrid\64bit\Vapoursynth\Lib\site-packages\vsdeoldify\models\checkpoints"

I attached the new version of the filter.

Dan

So if they are downloaded beforehand, there should be no need to download them, right?