Selur's Little Message Board

Thank you Mr. Selur, but I can't find any good information, I don't understand well the changes you have made from one version to another of the Hybrid program so that one version works for me and another doesn't. Anyway, thank you very much for your patience and forgive my way of expressing myself because I am from Spain and I don't understand English.

If my assumption correct, that your script does convert HDR to SDR for dolby vision content you should disable HDR signaling in the x265 settings.

Also, does the output of your script work as intended if you look at it in Avspmod or similar tools?

No, my script does not convert HDR to SDR. The input is HDR and the output is also HDR.

UHD Blu-rays include two types of Dolby Vision: MEL and FEL.
MEL is an "empty" video (no image) that simply contains the Dolby Vision metadata (RPU), and FEL is a video that in addition to containing the metadata (RPU) also shows image and to correctly encode the FEL video you must also process that second video and "bake in" the FEL into the BL video (the main HDR video).

There are several plugins that do this, for example:
- MapNLQ: https://github.com/Asd-g/AviSynthPlus-MapNLQ
- DoViBaker: https://github.com/erazortt/DoViBaker

If you want to encode videos with Dolby Vision coming from UHD Blu-rays it is essential and necessary to use these plugins, otherwise the resulting video will not be the right one.


And in short, the script I use in Hybrid uses the first plugin, that is MapNLQ.




I have some questions that maybe you could help me to understand, if you know something I would be very grateful:

- 1
As I told you in previous messages, I have been doing encodes in Hybrid for almost 2 years using these plugins (I used to use DoViBaker and now I use MapNLQ as it is better optimized and faster, but both perform the same function) and I have never had any problem.
I don't know if I have been doing it correctly during these last years, but what I have always done is simply to set Avisynth 64bit in Hybrid (that is : Config > Internals > Avisynth type: 64bit) and then in Hybrid I click on the open file option and instead of choosing an MKV file as I would normally do, I choose an avs file (which contains the script of MapNLQ or DoViBaker or whatever)... and that's it, then I perform an x265 encode as I would normally do as if it were an MKV. Is that correct?


By the way, I have Avisynth+ installed on my PC: https://github.com/AviSynth/AviSynthPlus/releases
And this plugins pack: https://gitlab.com/uvz/AviSynthPlus-Plugins-Scripts1

And I use avs scripts in AvsPmod (https://github.com/gispos/AvsPmod/releases) and Hybrid without problems.
And I have never needed to read dependencies or anything inside the scripts, I have always just used the functions without loading anything and it has always worked perfectly for years.


The problem why I opened this thread, is that a few days ago I formatted my PC and reinstalled Windows and then I installed Hybrid (the current version) and when I tried to open an avs file as I have been doing for years it didn't work.
I couldn't get it to work, so I thought maybe it was a problem with this version of Hybrid, I don't know.... So I started to try older versions of Hybrid, and indeed I got to version 2024.05.05.1 and now everything was working correctly, now works the same as it has always worked for years.

So:
The latest version of Hybrid that loads avs scripts without problems is version 2024.05.05.1, in all versions after that I can't get it to work (version 2024.05.11.1 and later).
And in the Hybrid changelog I do not find anything that has changed in this regard or anything that has nothing to do with this apparently.

Would it be possible that the current version of Hybrid (or in future Hybrid updates) this will work again as it has always worked for years (no need to load dependencies or do anything weird)? Since I've been using it this same way for years and I've never had any problems, so this is a bit frustrating for me and I wouldn't want to be stuck forever on an old version of Hybrid.




- 2
And a second question:
Following on from this thread and from looking at the log texts you have highlighted for me, I usually read in them the text "709". For example in your last messages you have shared me this text (coming from my logs):

And then I was a bit surprised to find so many settings referring to the 709 color space, since I'm doing HDR encodes and therefore the color space should be 2020 and not 709.
So I'm a bit worried...
Why does it say 709 instead of 2020? Is this wrong? Or is this right and correct?

Does this mean that the source is being read as 709 instead of 2020 (even though later the output is correctly written as 2020)?
Because if it really is reading the source as 709 then I would guess that a sort of "funnel" or "bottleneck" is forming and then they are missing a portion of the color spectrum, since 2020 encompasses a wider breadth and range of colors than 709. correct? So my logic tells me that.



Best regards and thank you very much as always! 

I'm a bit confused since you wrote that you are doing this for years and then you write you recently changed to MapNLQ.
But okay, since you wrote your script with MapNLQ works with 2024.05.05.1:
1. Does the Avisynth Preview work with that version? Dreate a debug output of you opening that script in Hybrid in Avisynth Preview.
2. Create (another) debug output of Hybrid processing/encoding that script.

Quote:Why does it say 709 instead of 2020? Is this wrong? Or is this right and correct?

The output vui values are set in the tab of the encoder, by guess is that atm. Hybrid takes the values it guesses these from the input.
You need to adjust these for HDR content. 709 is wrong for HDR content.
Yes, 709 can't represent all the colors HDR can.

Cu Selur

First of all, thank you very much again Mr. Selur.

(26.08.2024, 04:52)Selur Wrote: [ -> ]I'm a bit confused since you wrote that you are doing this for years and then you write you recently changed to MapNLQ.

Yes, I have been using avs scripts in Hybrid for years (not only I have used these 2 plugins, I have also used other plugins).
And specifically about DoViBaker and MapNLQ plugins: DoViBaker I started using almost 2 years ago, and MapNLQ I've been using for the last 9 months (and as I explained before, during these last months I use MapNLQ instead of DoViBaker, so I don't use DoViBaker anymore).


And as I told you before, everything has always worked perfectly for me without problems. But in the latest versions (2024.05.11.1 and later) it doesn't work.
I guess if you go edit the scripts to manually load all the dependencies as you indicated to me days ago maybe it would work... but that would be a bit annoying and quite cumbersome and laborious, and during all these years it has never been necessary to do any of this (versions 2024.05.05.1 and earlier), before it just worked without complications.

That's why I asked you in my previous message the following:

(25.08.2024, 21:22)murriato Wrote: [ -> ]Would it be possible that the current version of Hybrid (or in future Hybrid updates) this will work again as it has always worked for years (no need to load dependencies or do anything weird)? Since I've been using it this same way for years and I've never had any problems, so this is a bit frustrating for me and I wouldn't want to be stuck forever on an old version of Hybrid.

(26.08.2024, 04:52)Selur Wrote: [ -> ]But okay, since you wrote your script with MapNLQ works with 2024.05.05.1:
1. Does the Avisynth Preview work with that version? Dreate a debug output of you opening that script in Hybrid in Avisynth Preview.
2. Create (another) debug output of Hybrid processing/encoding that script.

With Avisynth Preview you mean Hybrid > Tools > avsviewer, correct?

In that case, with the latest versions of Hybrid (2024.05.11.1 and later) it does not work, it shows this error:


But with older versions of Hybrid (versions 2024.05.05.1 and earlier) it works perfectly:





By the way, I have tested with CMD the avsInfo64 that is included inside Hybrid and it also works without problems in older versions of Hybrid:





I attach the two debug outputs, but I think that avsviewer does not generate any text in the log (because I open the avsviewer tool and open the avs script and in the HybridDebugOutput.txt file no new text is written when I perform these steps.

(26.08.2024, 04:52)Selur Wrote: [ -> ]

Quote:Why does it say 709 instead of 2020? Is this wrong? Or is this right and correct?

The output vui values are set in the tab of the encoder, by guess is that atm. Hybrid takes the values it guesses these from the input.
You need to adjust these for HDR content. 709 is wrong for HDR content.
Yes, 709 can't represent all the colors HDR can.

Cu Selur

Yes, I know that 709 is SDR and 2020 is HDR.

My doubt was another one, not that one, let me be more specific and explain myself better:
The encodes I have done for years in HDR in Hybrid I have always obtained a correct output, that is to say the output has always been HDR... the MediaInfo of the encoded outputs are always HDR (BT.2020, PQ, MaxFALL/MaxCLL values, etc).


But as a result of this thread and the problem I have, I have had to review my logs so I can attach them to you in case they are helpful.
And I have noticed that in these logs in many occasions the text "709" appears, for example here:


So reading in the log so many "709" has surprised me, and I'm worried because I'm encoding HDR content and not SDR, so if I'm encoding HDR it should not appear in the log so many settings that refer to the 709 color space, should all appear as 2020.
Is this correct (should always appear 2020 and should never appear 709)?
Or do these log texts not matter and do not imply anything and I should not worry?


My encodes (my output videos that I generate with Hybrid) are all correct and all are HDR output.
But I'm worried because reading this text fragment in the log I get the feeling (or at least that's what my logic says) that the source is being read as 709 instead of 2020 (although the output is correctly written as 2020)... and in case this is really happening then it will be forming a kind of funnel/bottleneck causing a part of the color spectrum to be lost, since 2020 encompasses a wider breadth and range of colors than 709.



Thank you very much as always!





P.S.
Other things I discovered when I was testing this:

1.
I noticed that in the Log it is written that it cannot find color matrix and then it assumes it is bt709 since the video size is larger than 720*576.



Then I thought that maybe this is the problem... so I edited the avs script and removed the z_Spline36Resize line (so that the video is still 2160 instead of resized to 1080).
And indeed now in the Log it is written that it cannot find color matrix and then it assumes it is bt2020ncl since the video size is larger than 1920*1088 (although it should be 1080 instead of 1088, right?).



But this does not produce any change, the results remain the same.
So this seems to be indifferent and does not affect anything.



2.
As I told you Hybrid is not able to read the color of the source and then adjusts it as bt709, but now I have seen that the same thing happens with the framerate, it is also not able to read it well and adjusts it wrong as it configures it as 23976/1000 instead of 24000/1001.






And by the way, I noticed that the output encoded videos are actually bt709 and not bt2020 as I told you before... before this did not happen to me because I had manually set the MKV tags here (all tags, matrix/range/transfer matrix/primaries/etc): Config > MKV tagging > VUI > Matroska advanced video signaling


But I think this is not correct, Hybrid should not read these videos as bt709 and should not have these problems.

For example when I encode a mkv input (instead of avs script) I don't have any of these problems and I don't need to use MKV tagging (VUI > Matroska advanced video signaling).



3.
I noticed that the Auto crop option does not work when the input is an avs script.
That is, when the input is an MKV file you can press that option and Hybrid tries to guess the necessary crop, but when the input is an avs script you press that option and nothing happens.
I mean this:





4.
These problems (not detecting color, framerate, etc) I have noticed that they have nothing to do with MapNLQ.
Because if I make a very simple avs script, just with this line and nothing else:

I get all these same problems and the same results.

So these problems are using avs scripts in general, no matter what text I have written inside the script.

Okay, so instead of staying with one issue until it is fixed you start talking about other problems,....

Regarding 1:
Okay, so what changed is that in 2024.05.05.1 Hybrid loaded the script through import, while in 2024.05.11.1 it tries to load the script directly.
Which makes sense since no additional processing in Hybrid is done.
Problem is: The script is not compatible with Hybrid since it does not load its dependencies.
=> The proper way is to fix your script, to load the dependencies it uses.
As a workaround it might also work to do additional filtering in Hybrid, since this should trigger Hybrid to create a wrapper script.
BUT, this will not fix your second problem (2.).

Regarding 2:
Hybrid uses avsInfo to analyse .avs and MediaInfo to analyse .mkv input for the color matrix.
In the script you used:
You commented out the lines which set the color matrix and return the clip.
So this clip has no color matrix info and the only thing possible to do is: guess

That said, current avsInfo does not analyse the matrix info.
So if you want this to be fixed, create a sample script with sample file and all the dependencies used.
The script:
a. needs to load the dependencies
b. the script needs to set the color matrix and return a clip

This way I can reproduce the problem and try to adjust avsInfo to read the color matrix.

Regarding 3:
The autocrop uses ffmpeg or mplayer, my guess is they have problems with the script.
=> like with 1.&2. I need a working script with dependencies to reproduce the problem.

Regarding 4:
Yes, avsInfo does not look for the color matrix info.

Cu Selur

Ps.: The proper way to overwrite the input values would be to use "Filtering->Misc->Overwrite input"
PPs.: I'm not sure avisynth even signals the color matrix on the output. (I suspect it does not, but I'll try to find some info on this.)
PPPs.: You can try to open your script with avsViewer to check whether the preview works.

Reading the color matrix info from Avisynth scripts seems not possible since the output of Avisynth scripts does not contain this information.
see: https://avisynthplus.readthedocs.io/en/l...oInfo.html and http://avisynth.nl/index.php/Filter_SDK/.../VideoInfo

=> nothing Hybrid can do about that

Cu Selur

Looked at the crop detect and Hybrid properly uses:
to detect the crop values, but FFmpeg always reports that there is no cropping.
For a 720x480 Avisynth source ffmpeg will report something like:
When using '-vf cropdetect' instead of '-vf cropdetect=24:2', thus not limiting the black color detection or the mod, it seems to work and ffmpeg reports:
-> will adjust Hybrid in next dev

So:

• the crop on Avisynth scripts should work now.
  
• Color matrix detection of Avisynth input is not possible, since Avisynth doesn't have an interface to check the color matrix. You need to overwrite the input color matrix if the resolution based guess doesn't help.
  
• Would need a proper Avisynth script (which does not rely on autoloading) and files as sample to see if there is an issue with the Avisynth processing.
  

Cu Selur

Rewrote the Avisynth handling a bit and uploaded a new dev.
Try how that version works with your setup.
General advice: learn Avisynth if you feed Hybrid Avisynth scripts.
Blindly using stuff, like you do atm., will only cause problems in the future.

Cu Selur

(31.08.2024, 07:08)Selur Wrote: [ -> ]Hybrid uses avsInfo to analyse .avs and MediaInfo to analyse .mkv input for the color matrix.
In the script you used:

Code:

MKV="C:\Users\cebol\Desktop\ENCODES\HYBRID\Los juegos del hambre - Balada de pajaros.mkv"

BL=FFVideoSource(MKV).z_ConvertFormat(bit_depth=16).libplacebo_Tonemap(src_csp=3, dst_csp=1).z_ConvertFormat(pixel_type="YUV420P16", chromaloc_op="top_left=>top_left", resample_filter="spline36")
EL=FFVideoSource(MKV.ReplaceStr(".mkv", "_EL.mkv"))

MapNLQ(BL, EL)
z_ConvertFormat(pixel_type="YUV420P10", dither_type="error_diffusion")


z_Spline36Resize(dither="error_diffusion", 1920, 804, 0, 276, -0, -276)

#Prefetch(2)

SelectRangeEvery(8000,10)

#z_ConvertFormat(colorspace_op="2020ncl:2020:2020:limited=>2020ncl:2020:2020:limited", dither_type="error_diffusion")


#return last

You commented out the lines which set the color matrix and return the clip.
So this clip has no color matrix info and the only thing possible to do is: guess

That said, current avsInfo does not analyse the matrix info.
So if you want this to be fixed, create a sample script with sample file and all the dependencies used.
The script:
a. needs to load the dependencies
b. the script needs to set the color matrix and return a clip

This way I can reproduce the problem and try to adjust avsInfo to read the color matrix.

(31.08.2024, 07:16)Selur Wrote: [ -> ]Reading the color matrix info from Avisynth scripts seems not possible since the output of Avisynth scripts does not contain this information.
see: https://avisynthplus.readthedocs.io/en/l...oInfo.html and http://avisynth.nl/index.php/Filter_SDK/.../VideoInfo

=> nothing Hybrid can do about that

Cu Selur

(31.08.2024, 09:00)Selur Wrote: [ -> ]So:

• Color matrix detection of Avisynth input is not possible, since Avisynth doesn't have an interface to check the color matrix. You need to overwrite the input color matrix if the resolution based guess doesn't help.
  

Ps.: The proper way to overwrite the input values would be to use "Filtering->Misc->Overwrite input"

The lines that adjust the matrix color that you have indicated that I have commented out/disabled, in reality those lines should not be present.
I had the lines there because I was testing. I thought that maybe Hybrid didn't detect the matrix color because the avs script didn't have the video with it present, so then I was trying to manually assign the matrix color (with z_ConvertFormat and other plugins) to see if this way Hybrid was able to recognize the matrix color... but it didn't.
So that commented line is actually missing and shouldn't be there. It was just tests I was doing and then I forgot to delete it.

But well, you have already discovered that the output of the Avisynth scripts do not contain this information, so that's why Hybrid never recognized it even trying to put it in manually.


I didn't know the option: "Filtering->Misc->Overwrite input".
That's very interesting. So if I just manually set the option here as Rec.2020ncl it would be all correct?

(31.08.2024, 07:08)Selur Wrote: [ -> ]The autocrop uses ffmpeg or mplayer, my guess is they have problems with the script.
=> like with 1.&2. I need a working script with dependencies to reproduce the problem.

(31.08.2024, 09:00)Selur Wrote: [ -> ]Looked at the crop detect and Hybrid properly uses:

Code:

ffmpeg -i "PATH TO avs" -vf cropdetect=24:2 -f null NUL

to detect the crop values, but FFmpeg always reports that there is no cropping.
For a 720x480 Avisynth source ffmpeg will report something like:

Code:

[Parsed_cropdetect_0 @ 0000026d6fa61c80] x1:0 x2:719 y1:0 y2:479 w:720 h:480 x:0 y:0 pts:2 t:0.066733 limit:24.000000 crop=720:480:0:0

When using '-vf cropdetect' instead of '-vf cropdetect=24:2', thus not limiting the black color detection or the mod, it seems to work and ffmpeg reports:

Code:

[Parsed_cropdetect_0 @ 000002a154381c80] x1:4 x2:715 y1:0 y2:479 w:704 h:480 x:8 y:0 pts:2 t:0.066733 limit:0.094118 crop=704:480:8:0

-> will adjust Hybrid in next dev

So:

• the crop on Avisynth scripts should work now.
  

I tried to test the Auto crop option to see if it now works with avs scripts, but I can't use it.
The option appears disabled and you can't interact with it. And this happens all the time, both with avs scripts and MKV files.

(31.08.2024, 10:05)Selur Wrote: [ -> ]Rewrote the Avisynth handling a bit and uploaded a new dev.
Try how that version works with your setup.
General advice: learn Avisynth if you feed Hybrid Avisynth scripts.
Blindly using stuff, like you do atm., will only cause problems in the future.

Cu Selur

Now the encodes are already shown as 2020 in MediaInfo instead of 709. So good news!
And also the avs script works correctly without the need to read dependencies, as it was always the case in older versions.

There is one strange thing though:
As I told you days ago, when I open in Hybrid an avs script that has already inside a resize to 1080 then Hybrid reported that it cannot find color matrix and then it assumes it is bt709 since the video size is larger than 720*576.

And now obviously the same thing is still happening, but the strange thing is that even though this happens (Hybrid assumes the color as bt709) if I make an encode and then open it in MediaInfo it turns out that the encode is bt2020 anyway.
That is, it doesn't matter if Hybrid assumes that the matrix color is bt709 or bt2020ncl, because anyway it always generates a resulting encode that is 2020.

Is this normal?



It should be noted that there is a strange thing that keeps happening as I explained in previous messages, and I do not understand why.
And that is that I have checked the text of the logs after doing encodes and it still continues to appear many times the text "709" by numerous places, despite the fact that I am encoding 2020.

Is this a bit strange? Or is it normal and nothing to worry about?

I am referring to what I told you in previous messages:

(31.08.2024, 02:37)murriato Wrote: [ -> ]But as a result of this thread and the problem I have, I have had to review my logs so I can attach them to you in case they are helpful.
And I have noticed that in these logs in many occasions the text "709" appears, for example here:

Code:

"C:\Program Files\Hybrid\64bit\ffmpeg.exe" -y -loglevel fatal -noautorotate -nostdin -threads 8 -i "C:\Users\cebol\Desktop\ENCODES\MapNLQ.avs" -an -sn -vf format=yuv420p10le,scale=1920:1038 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range tv -pix_fmt yuv420p10le -strict -1 -vsync 0 -sws_flags spline -f yuv4mpegpipe

So reading in the log so many "709" has surprised me, and I'm worried because I'm encoding HDR content and not SDR, so if I'm encoding HDR it should not appear in the log so many settings that refer to the 709 color space, should all appear as 2020.
Is this correct (should always appear 2020 and should never appear 709)?
Or do these log texts not matter and do not imply anything and I should not worry?


My encodes (my output videos that I generate with Hybrid) are all correct and all are HDR output.
But I'm worried because reading this text fragment in the log I get the feeling (or at least that's what my logic says) that the source is being read as 709 instead of 2020 (although the output is correctly written as 2020)... and in case this is really happening then it will be forming a kind of funnel/bottleneck causing a part of the color spectrum to be lost, since 2020 encompasses a wider breadth and range of colors than 709.

In the current logs of this new version of Hybrid, for example, I can find these texts in which "709" is mentioned (although there are dozens more):

Quote:2024.09.12 - 02:44:22_Windows 11 Version 23H2 (64bit)_2024.09.01.1 - level 9: tokenizeAndStart(1): "C:\Program Files\Hybrid\64bit\ffmpeg.exe" -y -loglevel fatal -noautorotate -nostdin -threads 8 -i "C:\Users\cebol\AppData\Local\Temp\encodingTempSynthSkript_2024-09-12@02_44_22_6910_0.avs" -an -sn -color_primaries bt2020 -color_trc smpte2084 -colorspace bt709 -color_range tv -pix_fmt yuv420p10le -strict -1 -vsync 0 -f yuv4mpegpipe -

Quote:2024.09.12 - 02:44:22_Windows 11 Version 23H2 (64bit)_2024.09.01.1 - level 9:  argumentString(1): -y -loglevel fatal -noautorotate -nostdin -threads 8 -i "C:\Users\cebol\AppData\Local\Temp\encodingTempSynthSkript_2024-09-12@02_44_22_6910_0.avs" -an -sn -color_primaries bt2020 -color_trc smpte2084 -colorspace bt709 -color_range tv -pix_fmt yuv420p10le -strict -1 -vsync 0 -f yuv4mpegpipe -

2024.09.12 - 02:44:22_Windows 11 Version 23H2 (64bit)_2024.09.01.1 - level 9:  arguments(1):
  -y
  -loglevel
  fatal
  -noautorotate
  -nostdin
  -threads
  8
  -i
  C:\Users\cebol\AppData\Local\Temp\encodingTempSynthSkript_2024-09-12@02_44_22_6910_0.avs
  -an
  -sn
  -color_primaries
  bt2020
  -color_trc
  smpte2084
  -colorspace
  bt709
  -color_range
  tv
  -pix_fmt
  yuv420p10le
  -strict
  -1
  -vsync
  0
  -f
  yuv4mpegpipe
  -

I attach several debug outputs of some encode tests, in which you can search and find many "709":
- HybridDebugOutput - MKV (4K): Encode of a 4K MKV (the input is a normal mkv, not an avs script).
- HybridDebugOutput - avs script (4K): Encode of an avs script (the original 4K is kept, without resizing to 1080)
- HybridDebugOutput - avs script (4K resize to 1080): Encode of an avs script (resize to 1080 within the avs script itself)




By the way, when I encode in this version of Hybrid the result ends up with 2 files (.265 and .mkv) instead of only .mkv as it has always happened to me.
I guess it's because this option is disabled as well as the Auto crop option. And I have always chosen MKV in this option but now I can't choose anything, and I guess this is the cause of this problem, but I don't know.







Thank you very much Mr. Selur as always!