Deoldify Vapoursynth filter

[Dan64]

not at home atm. will look at it later and report back, had a quick look at beginning of 'The Trial'. Sadly, as soon as there is motion, things turn b&w.

In effect DeepRemaster provides less colored frames, but for some movies, this characteristic can been used to retain the visual impact of original black and white version of the movie. It is possible to mitigate this effect with the parameter "Vivid" and/or by enabling RefMerge. Try also the samples provided using DeepRemaster.

Dan

[Selur]

Back at home,...
Reading that post, I'm really thinking about dropping support for HAVC in total.

I think that is time to rename the filter DeOldify in HAVC as shown in the following picture

I'll rename it to 'DeOldify/HAVC'. Not starting to really rename the filter since then I would also have to rename all the existing (~60) parameters for consistency.

• vsUseDeOldify
  
• vsDeOldifyResizeChroma
  
• vsDeOldifyDeviceIndex
  
• vsDeOldifyRenderFactor
  
• vsDeOldifyModel
  
• vsDeOldifyDDCCombine
  
• vsDeOldifyCombineStack
  
• vsDeoldifyCMCChromaThreshold
  
• vsDeOdifyLMMLuma
  
• vsDeOdifyLMMWhite
  
• vsDeOdifyLMMSaturation
  
• vsDeOldifyChromaSmoothing
  
• vsDeOldifyChromaSmoothingValue
  
• vsDeOldifyChromaStabilizer
  
• vsDeOldifyChromaStabilizerValue
  
• vsDeOldifyDDColorModel
  
• vsDeOldifySat
  
• vsDeOldifyDDColorSaturation
  
• vsDeOldifyHue
  
• vsDeOldifyDDColorHue
  
• vsDeOldifyDDColorRenderFactor
  
• vsDeOldifyUseDark
  
• vsDeOldifyDarkValue
  
• vsDeOldifySimpleMergeWeight
  
• vsDeOldifyDDColorTweaks
  
• vsDeOldifyDDColorTweaksValue
  
• vsDeOldifyCombineSwitch
  
• vsDeOldifyDDColorFP16
  
• vsDeoldifyALMThresh
  
• vsDeOldifyALMExp
  
• vsDeoldifyALMWeight
  
• vsDeOldifyColorMap
  
• vsDeoldifyColorMapGeneral
  
• vsDeOldifyPreset
  
• vsDeoldifyColorFix
  
• vsDeOldifyDDFP16
  
• vsDeOldifyDDDebug
  
• vsDeOldifyColorTune
  
• vsDeOldifyVideoTune
  
• vsDeOldifyEnableDeepEx
  
• vsDeOldifySCThreshold
  
• vsDeOldifySCMinFreq
  
• vsDeOldifyDeepExPreset
  
• vsDeOldifyDeepExVivid
  
• vsDeOldifyDeepExMethod
  
• vsDeOldifySceneFramedir
  
• vsDeOldifyDeepExOnlyRefs
  
• vsDeOldifyDeepExRefMerge
  
• vsDeOldifyDeepExModel
  
• vsDeOldifyDeepExFrames
  
• vsDeOldifyDeepExRefWeight
  
• vsDeOldifyDeepExRefThresh
  
• vsDeOldifyColorMNetMode
  
• vsDeOldifyScThtSSIM
  
• vsDeOldifySCNormalize
  
• vsDeOldifyScThtOffset
  
• vsDeOldifyScMinInt
  
• vsDeOldifyColoringMethod
  
• vsDeOldifyAlternativeModel
  

(+ ~70 generated parameters to handle masking & co)

NEW USE of Method=5 "HAVC different_from_video"
....
In this case the clipRef must use the external video selected using "Ref FrameDir", and Hybrid need to provide the necessary conversion code to render clipRef compatible with HAVC.
"Ref FrameDir", and Hybrid need to provide the necessary conversion code to render clipRef compatible with HAVC.

So you whish Hybrid to create:

# ---------------- ADDING EXTERNAL COLORED VIDEO -------------------------
# Current color space: YUV420P8, bit depth: 8, resolution: 300x232, frame rate: 25fps, scanorder: progressive, yuv luminance scale: limited, matrix: 709, transfer: bt.709, primaries: bt.709, format: AVC
# the frame rate is converted to be the same of B&W clip (should not be relevant)
clipRef = core.lsmas.LWLibavSource(source="test_green_sample.mp4", format="YUV420P8", stream_index=0, cache=0, fpsnum=24000, fpsden=1001, prefer_hw=0)
# Resizing using bicubic spline to the same size of B&W clip
clipRef = core.fmtc.resample(clip=clipRef, kernel="spline16", w=920, h=672, interlaced=False, interlacedd=False)
clipRef = core.std.SetFrameProps(clipRef, _Matrix=vs.MATRIX_BT709)
if validate.transferIsInvalid(clip):
  clipRef = core.std.SetFrameProps(clip=clipRef, _Transfer=vs.TRANSFER_BT709)
if validate.primariesIsInvalid(clip):
  clipRef = core.std.SetFrameProps(clip=clipRef, _Primaries=vs.PRIMARIES_BT470_BG)
clipRef = core.std.SetFrameProps(clip=clipRef, _ColorRange=vs.RANGE_LIMITED)
clipRef = core.std.AssumeFPS(clip=clipRef, fpsnum=24000, fpsden=1001)
clipRef = core.std.SetFrameProps(clip=clipRef, _FieldBased=vs.FIELD_PROGRESSIVE)# progressive

clipRef = core.std.SetFrameProps(clip=clipRef, _ColorRange=vs.RANGE_FULL)
clipRef = core.resize.Bicubic(clip=clipRef, format=vs.RGB24, matrix_in_s="709", range_s="full")

Wouldn't it make more sense to add something like:

havc_refclip = core.bs.VideoSource(source="PATH_TO_FILE")
if clipRef.num_frames != clip.num_frames:
  raise vs.Error('HAVC: Reference clip and source clip frame count differ!')
  
if havc_refclip.format.id != vs.RGB24:
  refframe = havc_refclip.get_frame(0)
  if frame.props.get('_ColorRange', None) != vs.RANGE_FULL :
    clipRef = core.resize.Bicubic(clipRef, range_in_s="limited", range_s="full")  
  if frame.props.get('_Matrix', None) != vs.MATRIX_BT709 :
    clipRef = core.resize.Bicubic(clipRef, matrix_in_s="470bg", matrix_s="709")
  havc_refclip = core.resize.Bicubic(clip=havc_refclip, width=clip.width, height= clip.height, format=vs.RGB24, matrix_i_s="709", range_s="full")
clipRef = core.std.AssumeFPS(clip=clipRef, fpsnum=clip.fps_num, fpsden=clip.fps_den)

inside HAVC itself?

Hybrid will not start to analyse random reference clips with mediainfo, mplayer, ffmpeg,... like it does for normal inputs.
I can adjust Hybrid to add some generic code like above, but it seems like a bad idea to rely on tons of external code.
So you might want to think about whether this really makes sense, especially since you should do the color, resolution, frame count checks inside HAVC anyway to make sure the input is as expected.

I'll look into adding 'NEW ColorMNet encode_mode=2 -> "remote all-ref" ' and '5) NEW PRESET: BlackWhiteTune'.

Cu Selur

[Selur]

Changed checkbox label to 'DeOldify/HAVC', added 'ColorMNet encode_mode=2' and 'BlackWhiteTune'.
=> uploaded new Hybrid_deoldify


Cu Selur

[Selur]

I do not understand ref_freq.

ref_freq: int = None,
...
    :param ref_freq:            If (ref_merge > 0 or method == 5). If > 0 will be generated at least a reference frame
                                every "ref_freq" frames. range [0-1500]. If is not set, is assigned automatically a
                                value depending on ref_merge value and method.

I interpret this as that:
1. it should only be used if ref_merge > 0 or method == 5.
2. If it is set to 'None' depending on ref_merge value and method some automatic value is assigned.
3. If set to a value > 0 at least a reference frame every "ref_freq" frames will be generated (1-1500).
If set to 0 ???
Is there a difference between None and 0?
If there is no difference, then 'None' should not be used at all.
If there is a difference, please explain it.

Cu Selur

Ps.: updated Hybrid_deoldify to include 'Deep-Remaster' as ex_method. (did not add ref_freq since I don't get it)

[Dan64]

Back at home,...
Reading that post, I'm really thinking about dropping support for HAVC in total.

I think that is time to rename the filter DeOldify in HAVC as shown in the following picture

I'll rename it to 'DeOldify/HAVC'. Not starting to really rename the filter since then I would also have to rename all the existing (~60) parameters for consistency.
Wouldn't it make more sense to add something like:

havc_refclip = core.bs.VideoSource(source="PATH_TO_FILE")
if clipRef.num_frames != clip.num_frames:
  raise vs.Error('HAVC: Reference clip and source clip frame count differ!')
 
if havc_refclip.format.id != vs.RGB24:
  refframe = havc_refclip.get_frame(0)
  if frame.props.get('_ColorRange', None) != vs.RANGE_FULL :
    clipRef = core.resize.Bicubic(clipRef, range_in_s="limited", range_s="full") 
  if frame.props.get('_Matrix', None) != vs.MATRIX_BT709 :
    clipRef = core.resize.Bicubic(clipRef, matrix_in_s="470bg", matrix_s="709")
  havc_refclip = core.resize.Bicubic(clip=havc_refclip, width=clip.width, height= clip.height, format=vs.RGB24, matrix_i_s="709", range_s="full")
clipRef = core.std.AssumeFPS(clip=clipRef, fpsnum=clip.fps_num, fpsden=clip.fps_den)

inside HAVC itself?

Ok in not renaming vsDeoldify I was not aware that to change the name it would be necessary so many changes (for the same reasons I don't have changed the name of the project in GitHub).

Regarding the possibility to add an external video, it is not clear your proposal. 
If necessary I can add some code (but I prefer to don't use bs.VideoSource is still too slow), but I cannot implement all the transformations that you usually add when a clip is loaded in Hybrid.
In this case in the string sc_framedir need to be passed the path to the external movie.
Could you clarify better ?

Thanks,
Dan

---

I do not understand ref_freq.

ref_freq: int = None,
...
    :param ref_freq:            If (ref_merge > 0 or method == 5). If > 0 will be generated at least a reference frame
                                every "ref_freq" frames. range [0-1500]. If is not set, is assigned automatically a
                                value depending on ref_merge value and method.

I interpret this as that:
1. it should only be used if ref_merge > 0 or method == 5.
2. If it is set to 'None' depending on ref_merge value and method some automatic value is assigned.
3. If set to a value > 0 at least a reference frame every "ref_freq" frames will be generated (1-1500).
If set to 0 ???
Is there a difference between None and 0?
If there is no difference, then 'None' should not be used at all.
If there is a difference, please explain it.

Cu Selur

Ps.: updated Hybrid_deoldify to include 'Deep-Remaster' as ex_method. (did not add ref_freq since I don't get it)

In effect in the GUI there is already the field "SC min freq" and "SC thresh" which are currently used by HAVC_ddeoldify. Now these 2 parameters can to be passed also to HAVC_deepex.

ref_thresh = "SC thresh" 
ref_freq = "SC min freq"

 When I introduced the "Ref merge" you added also the field "Threshold", but this field is not really necessary because you can use "SC thresh". It is possible to do that because when "Ref merge" is set "SC min freq" is set equal to 1 and all the remaining scene detection parameters will be ignored by HAVC_dddeoldify, so that you can use  "SC min freq" and "SC thresh" for the "Ref merge", so that the parameter "Threshold" could be removed because is redundant. But I don't know your code so is up to you to decide what to implement.

Dan

[Selur]

Could you clarify better ?

• I will not start analyzing such clips, since it does not make sense.
  
• HAVC should check that the clip is RGB24 and has the same frame count and resolution as the source and either convert something compatible with the used clip by itself or throw an error.
  

If HAVC throws an error (and the documentation states that these criteria (resolution, color format, frame count) should be met), Hybrid could use this code:

# load clip
havc_refclip = core.bs.VideoSource(source="PATH_TO_FILE")
# validate frame count
if clipRef.num_frames != clip.num_frames:
  raise vs.Error(f"HAVC: Reference clip ({clipRef.num_frames}) and source clip frame count ({clip.num_frames}) differ!")
# convert if not RGB24
if havc_refclip.format.id != vs.RGB24:
  refframe = havc_refclip.get_frame(0)
  # when the _ColorRange flag is not set or 'limited', assume that it is 'limited' and convert to 'full'
  if frame.props.get('_ColorRange', None) != vs.RANGE_FULL :
    clipRef = core.resize.Bicubic(clipRef, range_in_s="limited", range_s="full")
  # when the _Matrix-flag isn't set or not '709' assume '470bg' and convert to '709'
  if frame.props.get('_Matrix', None) != vs.MATRIX_BT709 :
    clipRef = core.resize.Bicubic(clipRef, matrix_in_s="470bg", matrix_s="709")
  # resize and convert to RGB24
  clipRef = core.resize.Bicubic(clip=havc_refclip, width=clip.width, height= clip.height, format=vs.RGB24, matrix_i_s="709", range_s="full")
else:
   # make sure resolution is correct (if the aspect ratio of the source and the reference differ, matching will not work properly)
    clipRef = core.resize.Bicubic(clip=havc_refclip, width=clip.width, height= clip.height)
clipRef = core.std.AssumeFPS(clip=clipRef, fpsnum=clip.fps_num, fpsden=clip.fps_den

before feeding the clip to HAVC, but normally I would expect that HAVC would do this, or explicitly state which characteristics the clipRef should have.
So the question is: Should Hybrid do this, or do you add (something like) this to HAVC?

Cu Selur

[Selur]

In effect in the GUI there is already the field "SC min freq" and "SC thresh" which are currently used by HAVC_ddeoldify. Now these 2 parameters can to be passed also to HAVC_deepex.

ref_thresh = "SC thresh"
ref_freq = "SC min freq"

When I introduced the "Ref merge" you added also the field "Threshold", but this field is not really necessary because you can use "SC thresh". It is possible to do that because when "Ref merge" is set "SC min freq" is set equal to 1 and all the remaining scene detection parameters will be ignored by HAVC_dddeoldify, so that you can use "SC min freq" and "SC thresh" for the "Ref merge", so that the parameter "Threshold" could be removed because is redundant. But I don't know your code so is up to you to decide what to implement.

That does not make any more sense.

If

ref_freq = "SC min freq"

Why introduce a new parameter (ref_freq) then?
=> Adding a new parameter does not seem to make sense and from the sound of it either ref_thresh or sc_threshold should get removed too.

Cu Selur

[Dan64]

The scene change detection is usually performed by HAVC_ddeoldify. But there are 2 exceptions:

1) When is used "Ref merge". How "Ref merge" works is quite complex but is explained at page 18 of HAVC User Guide. In summary in this case HAVC apply the following strategy:

1.     all the clip frames are colored using DDColor and/or DeOldify (because the parameter SC min freq = 1)
   
2.     using the threshold specified (in this case 0.10) a subset of colored frames is sampled by the scene change detection algorithm and these frames are used as reference images for the selected exemplar-based model (in this case ColorMNet).
   
3.     the frames obtained at step 1 are merged with the weight specified (in this case 0.50) with the frames propagated by the selected exemplar-based model.
   

2) When is used method=5. In this case the reference clip is not generated by HAVC_ddeoldy because is used an external clip already colored and HAVC_deepex must know how to generate the reference frames.

So the scene detection parameters are necessary also to HAVC_deepex. I reduced the number to the minimum necessary to enable HAVC_deepex to perform such task.

Dan

[Selur]

Okay, then back to my questions:
a. What happens if ref_freq is set to 0.
b. Is there a difference between ref_freq None and 0?
If there is no difference, then 'None' should not be used at all.
If there is a difference, please explain it.

Cu Selur

[Dan64]

Ok, I understand that is not easy to use None in the GUI, also ref_tresh is None, but you pass always a value.

So you can pass ref_freq=0 if the user has not selected a different value.

If ref_freq = 0 (or None) will be applied the following default values:

   If ref_merge is enabled, ref_freq is set to 0
   If method=5 ref_freq is set to DEF_MIN_FREQ (=15)

It is necessary also to change the rule to set core.num_threads=1.
Currently num_threads=1 is set every time is checked "Reference frame only".
Now is necessary to set num_threads=1 when:  is checked "Reference frame only"  AND  (0 < SC SSIM thresh < 1 OR SC min int > 1)

Thanks,
Dan