a big difference between the source and target

[kingcrimsonster]

The bug is in the difference between the right preview window and the encoded file when the filter is 709 - 709 or it is off.

Load source Tablitsa.mpg.

1 variant
now set this options



Switch to "Color" and check "Colormatrix" - this automatically set 709 - 709.

Now press "avisynth preview. Both windows is the similar picture
If uncheck this filter - similar result
put the pipette on the green area in left and right window and measure RGB - 0, 162. 0



Encode this file, play it and put the pipette on the green area and measure RGB - 0, 193. 0



This is big difference between the right preview window and the encoded file.
If I want to manually adjust the color and I'll look at the right window the result will be incorrect.

2 variant
Now check "Picture Crop" option left-12, right-10, top-78, bottom-78 and set in "Picture resize" - target resolution 744
Switch to "Color" and check "Colormatrix" - this automatically set 709 - 601.
Now press "avisynth preview. Both windows is the big difference between the right and left preview.



put the pipette on the green area in left window and measure RGB - 0, 162. 0
put the pipette on the green area in right window and measure RGB - 0, 193. 0
Encode this file, play it and put the pipette on the green area and measure RGB - 0, 193. 0



in both variants there is correctly encoded file.

In 1 variant not correctly working preview, when no need convert colorspace. Please fix it.

[Selur]

Switch to "Color" and check "Colormatrix" - this automatically set 709 - 709.

Hybrid will change the ColorMatrix depending on resolution when 'Adjust by resolution' is enabled.
greater 720*576 -> Rec.709
less or equal 720*576 -> Rec.601


My guess atm is what happens is:
Hybrid analysis your source and since there is no colorMatrix or luminance info it guesses 709&tv.
Thus these values are used during the preview.
When ColorMatrix is disabled or the from&to values are the same, colorMatrix does nothing and the output colors will not be changed. As long as the guess Hybrid did regarding the colorMatrix&luminance was correct, the source is flagged correctly and the player which later plays the content is respecting the VUI values playback should show the same colors as the preview.
If the values are off or the player doesn't respect the VUI values, colors will look off.

A few questions:

• Is the colorMatrix properly set in the encoded file? What encoder did you use?
  
• Which preview did you use? Also which player?
  
• Is there a difference whether you use 'Always use Avisynth' or not?
  
• Does it make a difference if you use another 'Clipping' in the ColorMatrix settings value?
  
• Is there a difference when you use Vapoursynth?
  
• Is there a difference if you signal different colorMatrix values in in VUI section of the encoder? (keeping colorMatrix off!; If there isn't your player doesn't respect the color matrix values.)
  

In 1 variant not correctly working preview, when no need convert colorspace. Please fix it.

Will try to in case the above questions have been answered and there is something to fix.

Cu Selur

[kingcrimsonster]

Is the colorMatrix properly set in the encoded file? What encoder did you use?

Yes. x264

Which preview did you use? Also which player?

Avisinth Preview. PotPlayer

Is there a difference whether you use 'Always use Avisynth' or not?

No

Does it make a difference if you use another 'Clipping' in the ColorMatrix settings value?

No

Is there a difference when you use Vapoursynth?

No

Is there a difference if you signal different colorMatrix values in in VUI section of the encoder? (keeping colorMatrix off!; If there isn't your player doesn't respect the color matrix values.)

No

Hybrid will change the ColorMatrix depending on resolution when 'Adjust by resolution' is enabled.
greater 720*576 -> Rec.709
less or equal 720*576 -> Rec.601

My conclusion: Avisinth Preview working normally. Hybrid wrong interpreted color space for input files.
For variant 1:
Input file resolution is 720-576 This is Rec.601
Output file resolution with coorrected pixel aspect ratio to square 768-576 This is Rec.709
In "Color" tab I see "Colormatrix" from 709 to 709 with check "Adjust by resolution". This is wrong.
Should be from 601 to 709. Please fix it, when loaded Input file resolution is 720-576.
Similar error when use Vapoursynth

[Selur]

My conclusion:
Changing the colorMatrix flag doesn't change the way the colors are represented during preview in PotPlayer
-> PotPlayer or the decoder used doesn't respect the colorMatrix value set.

Input file resolution is 720-576 This is Rec.601

Not so easy. To my knowledge:
MPEG1 is always BT.601. MPEG2 default is BT.709, MPEG4 default is BT.601 in SD, MPEG4 default in HD is BT.709.
-> your content is MPEG-2 and thus should be BT.709 and this is what Hybrid assumes since no colorMatrix is specified.

[kingcrimsonster]

your content is MPEG-2 and thus should be BT.709 and this is what Hybrid assumes since no colorMatrix is specified

When I manual set is to bt601 - avisinth preview working correctly - result preview in right window corresponds to the final encoded file. In this case put the pipette on the green area in right window and measure RGB - 0, 193. 0.
Your quotes

MPEG2 default is BT.709

Hybrid will change the ColorMatrix depending on resolution when 'Adjust by resolution' is enabled.
greater 720*576 -> Rec.709
less or equal 720*576 -> Rec.601

One of these statements must be incorrect.
I'am open in the Hybrid MPEG2 NTSC file with resolution 720-480 - "Colormatrix" set it automatically "From BT601" - this is wrong?

Please, Selur, you yourself make tests of various files 720-480, 720-576 and make sure that the avisinth preview does not differ from the final encoded file.

greater 720*576 -> Rec.709
less or equal 720*576 -> Rec.601

In this case avisinth preview does not differ from the final encoded file.

[Selur]

I'am open in the Hybrid MPEG2 NTSC file with resolution 720-480 - "Colormatrix" set it automatically "From BT601" - this is wrong?

No that can be right, my guess is this source indicates the color matrix,... would need a debug output of the analysis.

Preview works fine for the samples I normally use.

[kingcrimsonster]

would need a debug output of the analysis.

[Selur]

The vob you used now is properly flagged as:

Color primaries                          : BT.601 NTSC
Transfer characteristics                 : BT.601
Matrix coefficients                      : BT.601

the previous file wasn't.

-> Problem is Hybrid guesses wrong since there is no indication that would allow it to guess right.

Cu Selur

Ps.: Unless your player respects the colorMatrix flag you will probably never know for sure where the problem is.

PPs.: If you want to read more into it, see: https://forum.doom9.org/showthread.php?t=133982