02.02.2025, 22:45
Hello Selur,
I will try to explain why I decided to resurrect DeepRemaster.
The main reason is that after having learned better about the exemplar-based models, I was able to include properly DeepRemaster in HAVC.
Respect to the version implemented previously I added the following improvements:
1) possibility to read the reference frames directly from the clip (my previous version was able to read only the reference images from a folder)
2) implemented a buffered read of the reference clip to don't slow down too much that start of the filter. Without this buffered read the start of the clip could take more than 5 minutes
4) implemented a feed strategy so that at DeepRemaster are privided 50% past reference frame and 50% future reference frames
While ColorMNet is working quite well, it has the problem that is not able to handle properly the future frames. In effect ColorMNet is not storing the full reference frame image (like DeepRemaster) but it store only the key points (e.g., representative pixels in each frame). This imply that the colored frames could have some colors that are very different from the reference image. DeepRemaster has not this problem since it store the full reference image. Unfortunately the number of reference images that DeepRemaster is able to use depends on GPU memory and power, because the time required for inference increase with the number of reference images provided.
The problem of gray images on fast moving scenes is a problem that affect mainly the exemplar-based models, especially DeepRemaster, because when there are fast moving objects in a clip is necessary to provide more reference images (for example 4/5 frames every second). ColorMNet has some interpolation capability while DeepRemaster is very basic and is unable to properly colorize a frame if is missing a reference image very similar.
Then there is the classical problem of gray noses or ears. This problem affect all the models, especially the examplar-based models. The reason is that despite the improvement of the technology. The current models for features recognition like ResNet50 are still trained using a frame of size 224x224. Using a so small resolution is very difficult to recognize a nose or ear.
You can find some info in this post: Questions regarding ColorMNet
I recently tried to colorize the movie Night of the living dead (1968). For this movie was available a small colored version: notld_201610
The problem was that the 2 movie were out of sync, the difference of about 400 frames was distributed randomly between them. And using ColoMNet (included in HAVC 4.6.8) by providing the colored version was a failure. So I implemented the version "remote all-ref" that theoretically should be able to handle reference frames not in sync with the clip to colorize. But even this attempt was a failure. The colored movie was quite good, but the color accuracy was lost. For example the the pink/magenta colored car seen at the beginning of the film was painted by ColorMNet in gray.
I was surprised when I discovered that DeepRemaster was able to handle perfectly well the reference frames not in sync and that at the same time was able to maintain very well the colors in the reference images.
At the following link: Night of the living dead (colorized, 1968)
Is available the movie colored using DeepRemaster. If you see some strange colors, is not a failure of DeepRemaster, but because were present in the reference frames (ColorMNet has not this problem, but the color accuracy is lost).
This is the main reason why I introduced the method = 5.
Dan
I will try to explain why I decided to resurrect DeepRemaster.
The main reason is that after having learned better about the exemplar-based models, I was able to include properly DeepRemaster in HAVC.
Respect to the version implemented previously I added the following improvements:
1) possibility to read the reference frames directly from the clip (my previous version was able to read only the reference images from a folder)
2) implemented a buffered read of the reference clip to don't slow down too much that start of the filter. Without this buffered read the start of the clip could take more than 5 minutes
4) implemented a feed strategy so that at DeepRemaster are privided 50% past reference frame and 50% future reference frames
While ColorMNet is working quite well, it has the problem that is not able to handle properly the future frames. In effect ColorMNet is not storing the full reference frame image (like DeepRemaster) but it store only the key points (e.g., representative pixels in each frame). This imply that the colored frames could have some colors that are very different from the reference image. DeepRemaster has not this problem since it store the full reference image. Unfortunately the number of reference images that DeepRemaster is able to use depends on GPU memory and power, because the time required for inference increase with the number of reference images provided.
The problem of gray images on fast moving scenes is a problem that affect mainly the exemplar-based models, especially DeepRemaster, because when there are fast moving objects in a clip is necessary to provide more reference images (for example 4/5 frames every second). ColorMNet has some interpolation capability while DeepRemaster is very basic and is unable to properly colorize a frame if is missing a reference image very similar.
Then there is the classical problem of gray noses or ears. This problem affect all the models, especially the examplar-based models. The reason is that despite the improvement of the technology. The current models for features recognition like ResNet50 are still trained using a frame of size 224x224. Using a so small resolution is very difficult to recognize a nose or ear.
You can find some info in this post: Questions regarding ColorMNet
I recently tried to colorize the movie Night of the living dead (1968). For this movie was available a small colored version: notld_201610
The problem was that the 2 movie were out of sync, the difference of about 400 frames was distributed randomly between them. And using ColoMNet (included in HAVC 4.6.8) by providing the colored version was a failure. So I implemented the version "remote all-ref" that theoretically should be able to handle reference frames not in sync with the clip to colorize. But even this attempt was a failure. The colored movie was quite good, but the color accuracy was lost. For example the the pink/magenta colored car seen at the beginning of the film was painted by ColorMNet in gray.
I was surprised when I discovered that DeepRemaster was able to handle perfectly well the reference frames not in sync and that at the same time was able to maintain very well the colors in the reference images.
At the following link: Night of the living dead (colorized, 1968)
Is available the movie colored using DeepRemaster. If you see some strange colors, is not a failure of DeepRemaster, but because were present in the reference frames (ColorMNet has not this problem, but the color accuracy is lost).
This is the main reason why I introduced the method = 5.
Dan