Optical flow using Hybrid -- Possible?

[markfilipak]

Hello Selur,

I have written a cmd script that performs 24000/1001fps-to-120fps optical flow with 1.001x audio & subtitle stretching. It works.
I played with Hybrid -- thank you! -- and its setup. It appears that Hybrid cannot do optical flow but I thought I'd ask, eh?  

The script does this:
1, FFprobe -- gather info
2, VapourSynth + InterFrame + SVPflow1+2 -- optical flow from 24000/1001fps to 120000/1001fps.
3, Pipe raw y4m to FFmpeg via 'VSPipe --y4m "FOO.VPY" - | ffmpeg -i pipe: -i "SOURCE" ...'.
4, FFmpeg -- force 120000/1001fps to 120fps without transcoding via '-filter_complex [0:v]settb=expr=1/120,setpts=expr=N[v]'.
5, FFmpeg -- stretch audio via '-filter_complex [1:a:#]atempo=1.001[a#]' for each audio stream, #.
6, MKVMerge -- stretch subtitles via '--sync #:0,1000/1001' for each subtitle stream, #.

The result is incredible.

I see how to stretch with Hybrid. Can Hybrid do the optical flow?

Warm Regards,
Mark.

[Selur]

Not sure what you mean by 'optical flow', but Hybrid can use Inteframe for frame interpolation.
Filtering->Vapoursynth/Avisynth->Frame->Interpolation->Frame Interpolation

Abount your script: no clue why you interpolate to 120000/1001fps instead directly to 120fps and then speed up to 120fps.

Cu Selur

[markfilipak]

Not sure what you mean by 'optical flow', but Hybrid can use Inteframe for frame interpolation.
Filtering->Vapoursynth/Avisynth->Frame->Interpolation->Frame Interpolation

Abount your script: no clue why you interpolate to 120000/1001fps instead directly to 120fps and then speed up to 120fps.

Cu Selur

Thank you for replying and for all the help you've given to so many people here and elsewhere. If you ever need help, I hope you will ask me.

Good questions.

I've not seen "optical flow" defined but from the pictures I've seen it appears to be,
1, object discrimination based on raw frame-to-frame pixel-block (4x4, 6x6, 8x8, etc.) pattern matching followed by,
2, motion vector (MV) generation followed by,
3, interframe interpolation using those generated MVs.
The optical flow process interpolates between decoded pictures as opposed to interpolating between the MVs found in the undecoded source frames (which, it appears, is never done because creating an entirely new transport stream by tiddling undecoded frames is too complicated or has never been attempted or has never been thought of).

I do 24000/1001fps-to-12000/1001fps interpolation because it's exactly 1-to-5.
I don't convert the 24000/1001fps source frames directly to 120fps for 2 reasons:
1, Instead of 1-to-5, the interpolation would be 1001-to-5000: not an integer and much too complex, and
2, After interpolation, I force the resulting 12000/1001fps to 120fps by changing the time base (TB) and the presentation time stamps (PTSs), only, not the number of frames. That reduces running time by T/1.001 thereby matching the speed and running time seen in movie theaters -- all without a 2nd transcoding.

If I may anticipate an additional question, I convert 24000/1001fps to 120000/1001fps instead of to 60000/1001fps because,
1, 1-to-5 conversion (to 120000/1001fps, a 5x interpolation) works/looks better than 2-to-5 conversion (to 60000/1001fps, a 2.5x interpolation), and
2, The final 120fps files are 1/8-th the size of the 60fps files -- amazing but true! -- and
3, 60Hz TVs simply drop the odd numbered frames -- 120fps to 60fps -- thereby completely avoiding judder -- no telecine!

Here is a simplfied example -- single-audio stream, single-subtitle stream -- of my workflow (in which the audio codec is dts and the subtitles are stream 3):

VSPIPE --y4m "FOO.VPY" - | FFMPEG -i pipe: -i "SOURCE" -filter_complex "[0:v]settb=expr=1/120,setpts=expr=N[v],[1:a:0]atempo=1.001[a0]" -map "[v]" -map "[a0]" -codec:v hevc_nvenc -preset 1 -rc 0 -spatial_aq 1 -bluray-compat 1 -multipass 2 -codec:a:0 dts -r 120 "TEMPTARGET.MKV"

MKVMerge --output "TARGET.MKV" --sync 3:0,1000/1001 "TEMPTARGET.MKV"

FOO.VPY is:

from vapoursynth import core
import havsfunc as havsfunc
oClip=core.ffms2.Source(source=r'SOURCE')
oClip=havsfunc.InterFrame(oClip, ...interframe.parameters.here...)
oClip=core.std.AssumeFPS(oClip, fpsnum=120000, fpsden=1001)
oClip.set_output()

Currently, the Windows cmd script is 443 lines and is totally automatic -- one click. It uses FFprobe to gather stream numbers and whatnot but I will probably probe via MediaInfo in the future.

Regarding interpolation, it appears Hybrid currently asks InterFrame to do frame repeating, only, not MV interpolation.

If you'd like to see the script, I'd prefer to show it via private message because it's still somewhat rough and I don't want people to use it until it's finished.

[Selur]

Regarding interpolation, it appears Hybrid currently asks InterFrame to do frame repeating, only, not MV interpolation.

You overlooked that.
Hybrid supports:

• FFmpegs framerate https://www.ffmpeg.org/ffmpeg-all.html#framerate (Filtering->Speed Change->Speed Settings->Interpolate frame rate)
  
• Vapoursynth Interframe/SVP https://www.svp-team.com/wiki/Manual:SVPflow (Filtering->Vapoursynth->Frame->Interpolation->Frame Interpolation->Interframe/SVP)
  
• Vapoursynth MVToolsFPS https://github.com/dubhater/vapoursynth-mvtools (Filtering->Vapoursynth->Frame->Interpolation->Frame Interpolation->MVToolsFPS)
  
• Vapoursynth RIFE https://github.com/HomeOfVapourSynthEvol...cnn-Vulkan (Filtering->Vapoursynth->Frame->Interpolation->Frame Interpolation->RIFE)
  
• Vapoursynth FrameRateConverter https://github.com/mysteryx93/FrameRateConverter (Filtering->Vapoursynth->Interpolation->Frame Interpolation->FrameRateConverter)
  
• Vapoursynth TorchRIFE https://github.com/HolyWu/vs-rife (Filtering->Vapoursynth->Interpolation->Frame Interpolation->TorchRIFE) (required torch addon)
  
• Avisynth SalFps3 http://avisynth.nl/index.php/SalFPS3 (Filtering->Frame->FPS adjust->SalFps3 )
  
• Avisynth Interframe/SVP https://www.svp-team.com/wiki/Manual:SVPflow (Filtering->Avisynth->Frame->FPS adjust->Interframe)
  
• Avisynth MVToolsFPSmod https://forum.selur.net/thread-1449-post...ml#pid8082 (Filtering->Avisynth->Frame->FPS adjust->MVToolsFPSmod)
  
• Avisynth FrameRateConverter https://github.com/mysteryx93/FrameRateConverter (Filtering->Avisynth ->Frame->FPS adjust->FrameRateConverter)
  

Cu Selur

[markfilipak]

... Hybrid can use Inteframe for frame interpolation.
Filtering->Vapoursynth/Avisynth->Frame->Interpolation->Frame Interpolation

Thank you. Yes, I see it now.

Instead of 'FPS: 119.8801', I'd prefer '5x'.

Hybrid's default overrides:
Super: {scale:{up:0},gpu:1,rc:true}
Vectors: {main:{search:{coarse:{distance:-8},type:2}},refine:[{thsad:250}]}
Smooth: {rate:{num:2,den:1,abs:false},algo:2,mask:{area:100},scene:{mode:0}}

Since I want 'perfect' and I don't care how long it takes, I will try:
Super: {pel:2,scale:{up:2,down:2},gpu:1,rc:true}
Vectors: {block:{w:8,h:4},main:{search:{coarse:{distance:-8},type:4}},refine:[{thsad:250}]}
Smooth: {rate:{num:5,den:1,abs:false},algo:23,mask:{area:100},scene:{mode:3}}

I'm aware of all the settings, but I must confess that I don't know what all of them mean.

Is there somewhere I can set this:
-codec:v hevc_nvenc -preset 1 -rc 0 -spatial_aq 1 -bluray-compat 1 -multipass 2
I'm looking at Base, 'Video: NVEnc'; NVEnc, 'Codec: h265' ... but I don't see a way to stuff the desired parameters above.

[Selur]

-codec:v hevc_nvenc -preset 1 -rc 0 -spatial_aq 1 -bluray-compat 1 -multipass 2

Hybrid does support NVEnc through NVEncC on Windows.
Using '-rc 0' = constant quantizer and '-spatial_aq 1' spatial adaptive quantization is not supported by Hybrid since it does not make sense trying to get fluctuating quantizers ins a contstatn quantizer mode.
'-multipass 2' should be the same as '-2pass' in ffmpeg iirc. Hybrid does only support this in variable bitrate encoding mode (since that to me is the only method where is makes sense).
About NVEncCs options see: https://github.com/rigaya/NVEnc/blob/mas...ions.en.md

Since I want 'perfect'

I doubt you will get that with Interframe since it's tuned for speed even when the settings force the slowest settings.

Cu Selur

[markfilipak]

Regarding interpolation, it appears Hybrid currently asks InterFrame to do frame repeating, only, not MV interpolation.

You overlooked ...

Indeed. Forgive me; I have so many balls in the air that I lost sight of the sun.  

Thanks for all the links.

I never used MVTools or AviSynth, and came directly to the VapourSynth race track, so I also suffer lack-of-experence.

---

Since I want 'perfect'

I doubt you will get that with Interframe since it's tuned for speed even when the settings force the slowest settings.

Yes, so I've been told. But I lack experience and only know InterFrame.

I've read that, in quality testing, InterFrame/SVP scored 3rd or 4th out of 4. I'd like to use Super & Vectors & Smooth directly (with?/without? InterFrame) but I've not discovered how to do that -- It's another of my balls in the air.  

[markfilipak]

Would you most recommend MVTools or AviSynth or AviSynth+? What are the developers' preferred?
I know that Hybrid is a frontend. I don't know of anyone who is as knowledgeable as you.

[Selur]

MVTools is available for Avisynth, Avisynth++ and Vapoursynth (https://github.com/dubhater/vapoursynth-mvtools).
Either Avisynth++ or Vapoursynth are the way to go, there is no mvtools without Avisynth/Vaporusynth/Avisynth++.
(as developer I would use Vapoursynth since it's Python with additional libraries and when writing a Vaporusynth script you are basically writing Python code )
For a start:
a. read the Avisynth (and Vapoursynth) documentation of the filters.
b. look in Hybrid\64bit\Avisynth\avisynthPlugins\MvToolsFPSmod.avs and look at the Vapoursynth script view when MVToolsFPS is used in Vapoursynth to get an example on how things connect.

libsvpflow1 and libsvpflow2 which are used in SVP are 'just' ports of MVTools which can use the GPU and are more restricted.
btw. FrameRateConverter (https://forum.doom9.org/showthread.php?t=174793) might be interessting.
Atm. it's still based on MVTools, but in the next coming versions it will also add support for RIFE for interpolation, which should help a lot.
(The thread itself is interresting to see what folks identify as problems during interpolation etc. )


Cu Selur

[markfilipak]

... (as developer I would use Vapoursynth since it's Python with additional libraries ...)

Ah! Thank you. I've never used AviSynth* but know and use Vapoursynth.

I'm developing a purely 'mechanical' approach to transcoding -- apparently, I'm a pioneer -- as opposed to 'cosmetics' like yadif. For example, for 30fps[24pps]: Detelecine to 24fps[24pps] + mv interpolation to 120fps[120pps], as opposed to 2x30fps[24pps] + yadif + bob to 120fps[120pps]. I'm having great success making incredible looking 120fps HEVC targets that are 1/8 the size of 24fps AVC sources. Because SVP is skewed to speed over quality, and because InterFrame (without your overrides) doesn't help and has vague parameters, I'm trying to use Super(), Analyse(), and SmoothFps() directly. Currently, I'm working on improving my understanding of the attributes in jsonSuper, jsonAnalyse, and jsonSmoothFps.



Here's an example of the sort of stuff I've posted to Doom9, Vapoursynth: https://forum.doom9.org/showthread.php?p...ost1953251




I tried to reproduce it here, but not even code tags preserve plain text.
Aside: Do you know of a way to force this board to use plain monospaced text? This so-called "rich text" makes it impossible to show what I want to show and it drives me crazy.

... FrameRateConverter (https://forum.doom9.org/showthread.php?t=174793) might be interessting.
Atm. it's still based on MVTools, but in the next coming versions it will also add support for RIFE for interpolation, which should help a lot.
(The thread itself is interresting to see what folks identify as problems during interpolation etc. )

Apparently, no one uses the motion vectors stored in the source frames. Do you know why?