"Limiter" in the "SmoothTweak"

[kingcrimsonster]

Not working "Limiter" in the "SmoothTweak" filter"

Limiter:
Clamp the output to 16-235 for luma & 16-240 for chroma

The screenshot shows that the output levels are 255. There should be a limit of no higher than 235



Sample

[Selur]

the Avisynth Hybrid uses looks fine to me:

SmoothTweak(limiter=true,TVrange=false,Lmode=2,HQ=true)

Looking at the preview Avisynth script:

LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\DGDecode.dll")
LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\SmoothAdjust.dll")
LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\TDeint.dll")
LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\TMM2.dll")
LoadPlugin("C:\PROGRA~1\Hybrid\32bit\AVISYN~1\nnedi3.dll")
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
# loading source: E:\!_Test\Test\Bars.mpg
#  input color sampling YV12
#  input luminance scale tv
Source = MPEG2Source(d2v="C:\Windows\Temp\mpg_9339b1eb3472a5fc531f647f4155cd52_853323747.d2v")
# current resolution: 720x576
# filtering
# color modifications
SourceFiltered = Source
SourceFiltered = SourceFiltered.SmoothTweak(limiter=true,TVrange=false,Lmode=2,HQ=true)
# deinterlace since the filter does not support interlaced content
Source = Source.AssumeTFF().TDeint(mode=1)
SourceFiltered = SourceFiltered.AssumeTFF().TDeint(mode=1)
# scaling to 768x576
Source = Source.BicubicResize(768,576)
SourceFiltered = SourceFiltered.nnedi3_rpow2(rfactor=2,cshift="GaussResize",fwidth=768,fheight=576)
# redeinterlaced source
Source = Source.AssumeTFF().Separatefields().Selectevery(4,0,3).Weave()
# redeinterlaced filtered source
SourceFiltered = SourceFiltered.AssumeTFF().Separatefields().Selectevery(4,0,3).Weave()
# stacking horizontal for filter preview
SourceFiltered = SourceFiltered.ConvertToRGB32(matrix="Rec709", interlaced=true)
Source = Source.ConvertToRGB32(matrix="Rec709")
StackHorizontal(Source, SourceFiltered)
PreFetch(6)
return last

this might be due to the color matrix:

string  matrix = "Rec601"

    Controls the colour coefficients and scaling factors used in RGB↔YUV conversions.

        "Rec601"  : Uses Rec.601 coefficients; scale full range [0d..255d] RGB ↔ TV range [16d..235d] YUV.
        "Rec709"  : Uses Rec.709 (HD) coefficients; scale full range RGB ↔ TV range YUV.
        "Rec2020" : Uses Rec.2020 (UHD) coefficients; scale full range RGB ↔ TV range YUV. AVS+
        "PC.601"  : Uses Rec.601 coefficients; keep range unchanged.
        "PC.709"  : Uses Rec.709 (HD) coefficients; keep range unchanged.
        "Average"  : Uses averaged coefficients (the luma becomes the average of the RGB channels); keep range unchanged.

source: http://avisynth.nl/index.php/Convert
that Hybrid used for the preview.
-> Have you looked at the histogram (in the normal Avisynth Preview), to check whether this is a problem of the preview or the processing?


Cu Selur

[kingcrimsonster]

Now I looked at the Avisint preview when turning on the histograms. The screenshot shows that the numerical values of the brightness limits (235) do not correspond to the actual brightness values in the Avisint preview window.





In the normal preview window actual brightness values is the 235

[Selur]

According to the histogram everything is fine, so my guess is that it's just a glitch in the preview.

[kingcrimsonster]

According to the histogram everything is fine, so my guess is that it's just a glitch in the preview.

Please remove this glitch!

[Selur]

Will do when I find the time to look further into it.
May be during the weekend.

Cu Selur

[kingcrimsonster]

I discovered a new glitch. Normal preview differs in levels from avisint preview. When the source file is played in the player, its levels are 0 - 255. When viewing the output file encoded in Hybrid, its levels are also 0 - 255. In the avisint preview window shows levels is 16 - 235 (Normal preview 30 - 218 - need to fix this) with all color filters disabled. As a result levels showed in the avisint preview and normal preview window (also digital signal levels when the “Histogram” button was pressed) do not correspond to the actual levels values output file encoded in Hybrid when this play in software and hardware players.
Previously avisint preview showed levels - the same levels were in the encoded file.
Inappropriate level values were shown by digital signal levels when the “Histogram” button was pressed. The screenshots in my previous post show that the white level is 255, but the digital value "Maximum" shows there are 235. Similarly with black level - 0 in avisint preview, 16 - digital signal levels minimum.
Now these values do not differ. But the signal levels in the avisint preview and the normal preview windows are very different from the levels in the encoded output file. Now it is impossible to properly control the level of the output signal.

[Selur]

I'll look into it.
Also the normal Preview is not intented for color accuracy can't do anything there, but will look at the avisynth preview.

Cu Selur

[Selur]

This will take some time.
Do you have color bar samples that are:
1. tv scale
2. pc scale
need some reference sample to check the changes against.
->Forget that I'll use Avisynths ColorBars option to check

Cu Selur

[Selur]

Argh, I think the default behavior Hybrid had was correct and me changing things was what broke it.

If you have a Yv12 source with tv range (16-235) and color matrix Rec.709 like your Bars.mpg
Histogram/ColorYUV analysis would should show 16-235, but the Preview which is RGB would show pc range (0-255) since.

"Rec709" : Uses Rec.709 (HD) coefficients; scale full range RGB ↔ TV range YUV.

Seems trying to grab the colors with a color picker from the preview without taking the color matrix into consideration is the problem.

Cu Selur