Deoldify Vapoursynth filter

[Dan64]

I forgot to add torch_dir parameter in HAVC_deepex().
The new RC6 should fix this issue.

Dan

[Selur]

That works.
Since this seems slower than the previous methods (untested, just a first impression), do you think ColorMNet should really be the default?
Will do some testing and create an adjusted Hybrid test version tomorrow after work. (going to bed in an hour or so since I have to get up tomorrow at 4am )

Cu Selur

[Selur]

Using:

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=0)

I get:

vapoursynth.Error: HAVC_deepex: encode_mode in (0, 1) and max_memory_frames == 0

but:

max_memory_frames:  Parameter used by ColorMNet model, specify the max number of encoded frames to keep in memory.
                                Its value depend on encode mode.
                                encode_mode=0, represent the batch size, suggested value are:
                                    6 or 8 : for 8GB GPU
                                    12 or 14 : for 12GB GPU
                                    24 or 28: for 24GB GPU
                                encode_mode=1, suggested values are:
                                    4 or 5 : for 8GB GPU
                                    8 or 9 : for 12GB GPU
                                    15 or 16: for 24GB GPU
                                encode_mode=2, there is no limit to this value, it can be set = 0 (all the frames in the clip)

=> confused Either the code or the documentation seems wrong

Cu Selur

[Dan64]

It is right, max_memory_frames is supported only with encode_mode = 2.
I attached a new RC7 with some bug corrected and an improved message (I hope).

I also attached a more serious sample, with more scripts for testing.

I will release a comparison of 2 models later.

Dan

[Selur]

It is right, max_memory_frames is supported only with encode_mode = 2.

Okay, then I do not understand what you mean with

:param DeepExMaxMemFrames: Parameter used by ColorMNet model, specify the max number of encoded frames to keep in memory.
Its value depend on encode mode.
encode_mode=0, represent the batch size, suggested value are:
6 or 8 : for 8GB GPU
12 or 14 : for 12GB GPU
24 or 28: for 24GB GPU
encode_mode=1, suggested values are:
4 or 5 : for 8GB GPU
8 or 9 : for 12GB GPU
15 or 16: for 24GB GPU

as I understand that text, if you have a 12 GB card and use encode_mode=0, it is recommended to use 12-14 frames.
But from your statement it sounds like for encode_mode 0 and 1 DeepExMaxMemFrames is always set automatically and needs to be specified for mode 0.
=> still confused

Using:

clip = HAVC_main(clip=clip)

coloring works.
Enabling 'Exemplar Models' and thus using:

clip = HAVC_main(clip=clip, EnableDeepEx=True, DeepExRefMerge=0, ScFrameDir="J:/tmp", DeepExModel=0, DeepExEncMode=0, DeepExMaxMemFrames=0)

I get:

Vapoursynth.Error: HAVC_deepex: max_memory_frames == 0 but encode_mode 2

But I used 'DeepExEncMode=0' not DeepExEncMode 2.

(Are you use this si a RC an not an alpha?)


Cu Selur
Ps.: Updated test version, but like mentioned it will crash,...

[Dan64]

I will change the text as following, I hope that is more clear.

DeepExMaxMemFrames:  Parameter used by ColorMNet model, specify the max number of encoded frames to keep in memory.
                                Its value depend on encode mode and must defined manually following the suggested values.
                                encode_mode=0, represent the batch size, suggested value are:
                                    6 or 8 : for 8GB GPU
                                    12 or 14 : for 12GB GPU
                                    24 or 28: for 24GB GPU
                                    must be > 0
                                encode_mode=1, suggested values are:
                                    4 or 5 : for 8GB GPU
                                    8 or 9 : for 12GB GPU
                                    15 or 16: for 24GB GPU
                                    must be > 0
                                encode_mode=2, there is no limit to this value, it can be set = 0 (all the frames in the clip)

If you know a way to get in python the amount of GPU memory I can try to set DeepExMaxMemFrames when is = 0, but for the moment it need to be always > 0. 
The only exception is for DeepExEncMode = 0 that has no memory limitation.

Please assign a value to max_memory_frame.

---

I released the new RC8. 
Now in the case max_memory_frames == 0, the value is automatically assigned with the values suggested depending on the total GPU memory.

Dan

[Selur]

You lost me.
This gets more and more confusing, since I have no clue how you envision this to work.

Now in the case max_memory_frames == 0, the value is automatically assigned with the values suggested depending on the total GPU memory.

Please, do not start to add additional stuff if you did not plan for it, like automatically setting the max_memory_frame value.
=> Remove that!

Please explain, ignore recommended values:

• If encode_mode = 0, what values are allowed for DeepExMaxMemFrames ? (min/max, independent of VRAM; taking vram into account is a user problem)
  
• If encode_mode = 1, what values are allowed for DeepExMaxMemFrames ? (min/max, independent of VRAM; taking vram into account is a user problem)
  
• If encode_mode = 2, what values are allowed for DeepExMaxMemFrames ? (min/max, independent of VRAM; taking vram into account is a user problem)
  

Please assign a value to max_memory_frame.

Okay, I will always set max_memory_frame/DeepExMaxMemFrames even if it's the default value.

Cu Selur

[Dan64]

I released the new RC9, with the following updated description

:param max_memory_frames:  Parameter used by ColorMNet model, specify the max number of encoded frames to keep in memory.
                                Its value depend on encode mode and must be defined manually following the suggested values.
                                encode_mode=0, represent the batch size, suggested value are:
                                    min=1, max=6/8    : for 8GB GPU
                                    min=1, max=12/14 : for 12GB GPU
                                    min=1, max=24/26 : for 24GB GPU
                                    If = 0 will be filled with the max value (depending on total GPU RAM available)
                                encode_mode=1, suggested values are:
                                    min=1, max=4/5    : for 8GB GPU
                                    min=1, max=8/9    : for 12GB GPU
                                    min=1, max=15/16  : for 24GB GPU
                                    If = 0 will be filled with the max value (depending on total GPU RAM available)
                                encode_mode=2, there is no limit to this value (it could be all the frames in the clip).
                                Suggested values are:
                                    min=250, max=10000
                                If = 0 will be filled with the value of 10000

To get the GPU memory available I use torch.cuda.mem_get_info() because what is relevant is the amount of GPU RAM seen by torch.

ColorMNet has huge memory problems when is used inside Vapoursynth. The only encode mode not having this limit is encode mode = 2 where max_memory_frames  can be set even above 10000.

The values are heuristic estimates based on my GPU that has 12GB of RAM (reported correctly by torch). In the case the frames are too high the encoding will fail with the error: GPU out of memory. 

There is nothing that I can do for that and in this case the user must lower the number of max_memory_frames defined in input to the filter.

Dan

[Selur]

According to that, max_memory_frames can always be set, independently of encode_mode.
Is that correct?

Assuming this is correct.
I will adjust Hybrid (tomorrow) to (independent of the encode_mode):
a. set min=0 and max = 10000
b. by default set 0
c. always set the max_memory_frames/DeepExMaxMemFrames value

Cu Selur

[Dan64]

Fine for me. 

This weekend I will be away and I will not be able to test new releases.

Try to see the new samples.

Try the encode mode=2 with a clip with more than 10000 frames and see what happens.
You will discover that:
   1) it is not possible to use the Preview
   2) you will not be able to estimate the fps from vspipe, because most of the time is spent by the script execution outside Vapoursynth.

I don't know if this behaviour could be a problem for Hybrid, please check.

Thanks,
Dan