[BUG] LSFmod in VapourSynth sets incorrect values

[The_Tinkerer]

Hope you are having a good holiday, I've got nothing to do but work with Hybrid and find bugs.

Looking at LSFMod script, from here --> https://github.com/HomeOfVapourSynthEvol...avsfunc.py

First, Lmode.

According to the above linked Vapoursynth script, Lmode only has positive values of 1-4, with 0 being none.

AviSynth seems to generate the correct Lmodes. But Vapoursynth does not. It has Lmode 5-7 (positive values) which should not exist. Selecting "Limit to over/undershoot on edges and to over/undershoot2 on not-edges" in Vapoursynth sets Lmode=7 when it should be Lmode=4. Setting Lmode to "Limit to over/undershoot" in VapourSynth sets Lmode=4 when it should set Lmode=1. Setting "No Limit" in Vapoursynth sets no Lmode, which implies default, but setting "No Limit" in AviSynth sets Lmode=0; in AviSynth, setting "Limit to over/undershoot" does **not** set Lmode=1, implying default. I'm not sure what the defaults actually are as the above linked script doesn't say, so maybe it should always get set?

4601  ### Lmode [int: ...,0,1,2,3,4]
4602  ### --------------------------
4603  ### Limit mode:
4604  ###    <0 : Limit with repair (ex: Lmode=-1 --> repair(1), Lmode=-5 --> repair(5)...)
4605  ###    =0 : No limit
4606  ###    =1 : Limit to over/undershoot
4607  ###    =2 : Limit to over/undershoot on edges and no limit on not-edges
4608  ###    =3 : Limit to zero on edges and to over/undershoot on not-edges
4609  ###    =4 : Limit to over/undershoot on edges and to over/undershoot2 on not-edges

Second, Smethod.

4533  ### Smethod [int: 1,2,3]
4534  ### --------------------
4535  ### Sharpen method: (only used in Smode=1,2)
4536  ###    =1 : 3x3 kernel
4537  ###    =2 : Min/Max
4538  ###    =3 : Min/Max + 3x3 kernel

Smethod is valid for Smode=2, but Vapoursynth does not set it when Smode=2... it only sets it when Smode=1. In both AviSynth and VapourSynth, Smethod=2 is **not** set, implying default, but LSFmod in AviSynth has more Smodes and Smethods (it includes older ones), so again, since the havsfunc.py script doesn't specify what the defaults actually are, it might be a good idea to specify them always.


So, in AviSynth, clicking the button to load "Slow" defaults and turning PreBlur on, gives:

LSFmod(Smode=5,Smethod=3,Lmode=4,preblur="ON",soft=-2,edgemaskHQ=true)

Doing the same in VapourSynth (I believe MinBlur(1) is the equivalent of PreBlur on in the AviSynth version) gives:

clip = havsfunc.LSFmod(input=clip, Smode=2, secure=True, Lmode=7, soft=-2, edgemaskHQ=True, preblur=1)

• Smode=2 is correct and is the same as Smode=5 in AviSynth
  
• Loading slow defaults in AviSynth selects Min/Max (enh) which sets Smethod=3; in Vapoursynth it selects Min/Max + 3x3 Kernel, which should set Smethod=3 but it doesn't. If it should be the same as AviSynth, then it should actually be Smethod=2, but if AviSynth is wrong and havsfunc.py is correct, then in AviSynth it should actually be setting Smethod=5 for Min/Max (enh) + 3x3 Kernel.
  
• secure=true is not set in AviSynth implying it's the default setting (not sure if that's correct or not); it's the opposite in VapourSynth, which if that's correct, then setting secure=true is necessary
  
• Lmode=7 is definitely wrong, it should be Lmode=4
  
• soft and edgemaskHQ settings are correct in both AviSynth and VapourSynth
  

Here's the list of what defaults should be for slow, from havsfunc.py:

4725  ### defaults="slow" : - strength    = 100
4726  ### ----------------- - Smode       = 2
4727  ###                   - Smethod     = 3
4728  ###                   - kernel      = 11
4729  ###
4730  ###                   - preblur     = -1
4731  ###                   - secure      = true
4732  ###                   - source      = undefined
4733  ###
4734  ###                   - Szrp        = 16
4735  ###                   - Spwr        = 4
4736  ###                   - SdmpLo      = 4
4737  ###                   - SdmpHi      = 48
4738  ###
4739  ###                   - Lmode       = 4
4740  ###                   - overshoot   = strength/100
4741  ###                   - undershoot  = overshoot
4742  ###                   - overshoot2  = overshoot*2
4743  ###                   - undershoot2 = overshoot2
4744  ###
4745  ###                   - soft        = -2
4746  ###                   - soothe      = true
4747  ###                   - keep        = 20
4748  ###
4749  ###                   - edgemode    = 0
4750  ###                   - edgemaskHQ  = true
4751  ###
4752  ###                   - ss_x        = Smode==3?1.00:1.50
4753  ###                   - ss_y        = ss_x
4754  ###                   - dest_x      = ox
4755  ###                   - dest_y      = oy
4756  ###
4757  ###

This is all I have checked so far so of course there could be more... 

---

If I'm reading the defaults correctly for "slow" then it should have ss_x=1.5 if Smode=2 or Smode=1 (?):

clip = havsfunc.LSFmod(input=clip, strength=100, Smode=2, Smethod=3, secure=True, Lmode=4, soft=-2, edgemaskHQ=True, preblur=1, ss_x=1.5)

[Selur]

To start please refer to https://github.com/Selur/VapoursynthScri...avsfunc.py
https://github.com/Selur/VapoursynthScriptsInHybrid contains the scripts that ship with Hybrid.

Also note that Avisynth and Vapoursynth filters often don't act the same way.

According to the above linked Vapoursynth script, Lmode only has positive values of 1-4, with 0 being none.

No.

### Lmode [int: ...,0,1,2,3,4]
### --------------------------
### Limit mode:
###    <0 : Limit with repair (ex: Lmode=-1 --> repair(1), Lmode=-5 --> repair(5)...)
###    =0 : No limit
###    =1 : Limit to over/undershoot
###    =2 : Limit to over/undershoot on edges and no limit on not-edges
###    =3 : Limit to zero on edges and to over/undershoot on not-edges
###    =4 : Limit to over/undershoot on edges and to over/undershoot2 on not-edges

says that negative values are limited by the corresponding repair(X) mode.
Which is why Hybrid, uses:

values.clear();
  values << QString("Repair(18)"); //index 0
  values << QString("Repair(17)");
  values << QString("Repair(16)");
  values << QString("Repair(15)");
  values << QString("Repair(14)");
  values << QString("Repair(13)"); //index 5
  values << QString("Repair(12)");
  values << QString("Repair(11)");
  values << QString("Repair(10)");
  values << QString("Repair(9)");
  values << QString("Repair(8)"); //index 10
  values << QString("Repair(7)");
  values << QString("Repair(6)");
  values << QString("Repair(5)");
  values << QString("Repair(4)");
  values << QString("Repair(3)");  //index 15
  values << QString("Repair(2)");
  values << QString("Repair(1)");
  values << QString("No Limit");
  values << QString("Limit to over/undershoot");
  values << QString("Limit to over/undershoot on edges and no limit on not-edges");
  values << QString("Limit to zero on edges and to over/undershoot on not-edges");
  values << QString("Limit to over/undershoot on edges and to over/undershoot2 on not-edges");
  this->addListField(QString("vsLSFModLmode"), QString("No Limit"), values, true);

That being said you are right, seems like the offset Hybrid uses is wrong atm. it uses -15, where it should use -18.
-> will fix.

It really would have helped if you had looked at the scripts that ship with Hybrid and report Avisynth and Vapoursynth separately.
Reading the rest will take me tons of time, since I will have to compare what is happening with the currently used scripts and first need to do a write up of your stuff to get to the bottom what is not working and what did change in the scripts since the time I implemented support for them.
-> will probably remove support for lsfmod in Avisynth and Vapoursynth until I find time and motivation to look into it.


Cu Selur

[The_Tinkerer]

Did not expect a reply today. Merry Xmas.

Believe it or not I was actually looking at C:\Program Files\Hybrid\64bit\vsscripts\havsfunc.py but I copy/pasted and used line numbers from Github, I guess so I could hotlink something. (Or in case the online is closer to what is in the latest dev version.)

But the actual comments I pasted were still the same. (This is why I'm not a programmer.)

Yeah this might take longer to fix as obviously it will take some time to look at.

Not in a rush, that's what the custom tab is for anyways, which I used, which worked.

[Selur]

Merry Xmas to you too.
For the future best keep Vapoursynth and Avisynth issues separated as they are totally different sections of the code which don't rely on each other and having to re-read a bug report multiple times to be sure which issue was where is a pain.
With a bit of luck I'll have some free time to look at it during next week. (no time this weekend due to friends&family)

Cu Selur

[The_Tinkerer]

I didn't write clearly when I said "Lmode only has positive values of 1-4, with 0 being none", I meant to say more like "The positive values in Lmode should be not go higher than 4".

I still haven't quite figured out how those 18 different "repair" modes are supposed to work within the context of the filter. I think they're supposed to be different degrees of grain removal (-18 being the most grain removed).

I also know it's not very good form to mix up different scripts / different versions of things in one bug report post, but I couldn't help but do some comparisons because I'm not sure in some cases which is supposed to be like the other.

[Selur]

Had some time and adjusted Lmode and SMethod, seems that fixed Vapoursynth.
-> send you a link via pm

Cu Selur