Selur's Little Message Board
Color change on encoding - Printable Version

+- Selur's Little Message Board (https://forum.selur.net)
+-- Forum: Hybrid - Support (https://forum.selur.net/forum-1.html)
+--- Forum: Problems & Questions (https://forum.selur.net/forum-3.html)
+--- Thread: Color change on encoding (/thread-3933.html)

Pages: 1 2 3

RE: Color change on encoding - Selur - 11.11.2024

For me Hybrid then uses the following command line:
ffmpeg -y -noautorotate -nostdin -threads 8 -i "G:\color_change_099.mkv" -map 0:0 -an -sn -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range tv  -pix_fmt yuv420p -fps_mode passthrough -vcodec ffv1 -coder 0 -context 0 -g 1 -level 1 -metadata encoding_tool="Hybrid 2024.11.10.1" "J:\tmp\2024-11-11@14_31_30_9810\color_change_099_reencoded_1_2024-11-11@14_31_30_9810_01.mkv"
which looks fine to me.
Comparing those two I used:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'F:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# loading plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/libhistogram.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
# Import scripts
import validate
# Source: 'G:\color_change_099.mkv'
# Current color space: YUV420P8, bit depth: 8, resolution: 1920x1080, frame rate: 23.976fps, scanorder: progressive, yuv luminance scale: limited, matrix: 709, transfer: bt.709, primaries: bt.709, format: FFV1
# Loading G:\color_change_099.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/color_change_099.mkv", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)
clip = core.hist.Classic(clip=clip)

reencode = core.lsmas.LWLibavSource(source="G:/color_change_099_reencoded.mkv", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)
reencode = core.hist.Classic(clip=reencode)


clip = core.std.Interleave([clip.text.Text("Org"),reencode.text.Text("Reencode")])

# output
clip.set_output()
to load the original and the reencode and then compare them and their histograms.

The output is identical. (also tried different histogram, but as expected they are identical)
Sorry, but I can't produce your problem and if I can't produce it, I can't fix it.
My guess is that, either you are doing something different or for some unknown reason whatever you used to compare these chose to convert them differently to rgb.

Cu Selur

RE: Color change on encoding - Lipomo - 11.11.2024

I ran many encodings of these episodes last night, and all have the issue.

Here is the debug log of one of the problematic videos : 

Created skript C:\Users\Admin\AppData\Local\Temp\encodingTempSynthSkript_2024-11-11@02_57_40_5510_0.vpy:---------------------
# Imports
import vapoursynth as vs
# getting Vapoursynth core
import ctypes
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("C:/Program Files/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll")
# loading plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/GrainFilter/RemoveGrain/RemoveGrainVS.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/CTMF/CTMF.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/TTempSmooth/TTempSmooth.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/DCTFilter.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DeblockFilter/Deblock/Deblock.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/BestSource/BestSource.dll")
# Import scripts
import psharpen
import havsfunc
import validate
# Source: 'G:\DB KAI part A et B\005-PartA\Davinci Resolve - CC\103.mkv'
# Current color space: YUV420P8, bit depth: 8, resolution: 1920x1080, frame rate: 23.976fps, scanorder: progressive, yuv luminance scale: limited, matrix: 709, transfer: bt.709, primaries: bt.709, format: FFV1
# Loading G:\DB KAI part A et B\005-PartA\Davinci Resolve - CC\103.mkv using BestSource)
clip = core.bs.VideoSource(source="G:/DB KAI part A et B/005-PartA/Davinci Resolve - CC/103.mkv", cachepath="C:/Users/Admin/AppData/Local/Temp/103_bestSource_2024-11-11@02_57_40_5510", track=0)
frame = clip.get_frame(0)
# setting color matrix to 709.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT709)
# setting color transfer (vs.TRANSFER_BT709), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT709)
# setting color primaries info (to vs.PRIMARIES_BT709), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT709)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED)
# making sure frame rate is set to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive
# denoising using MCTemporalDenoise
clip = havsfunc.MCTemporalDenoise(i=clip, settings="low", thSAD=400, thSAD2=400, thSCD1=400, thSCD2=100, truemotion=False, MVglobal=True, pel=4, pelsearch=5, search=5, searchparam=5, MVsharp=2, DCT=0, ncpu=1)
# sharpening using PSharpen
clip = psharpen.psharpen(clip=clip, strength=16.00)
# applying FineDeHalo to remove halos
clip = havsfunc.FineDehalo(clip, rx=2.50, ry=2.50, darkstr=0.00, brightstr=1.10)
# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# output
clip.set_output()
---------------------
ffmpeg started...
starting 2024-11-11@02_57_40_5510_02_video@03:03:38.014 - G:\DB KAI part A et B\005-PartA\Davinci Resolve - CC\Hybrid filtred finales\103.mkv
"C:\Program Files\Hybrid\64bit\ffmpeg.exe" -y -noautorotate -nostdin -threads 8 -f yuv4mpegpipe -i - -an -sn -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range tv -pix_fmt yuv420p -fps_mode passthrough -vcodec ffv1 -coder 0 -context 0 -g 1 -level 1 -metadata encoding_tool="Hybrid 2024.11.05.1" "C:\Users\Admin\AppData\Local\Temp\103_1_2024-11-11@02_57_40_5510_02.mkv"
2024-11-11@02_57_40_5510_02_video finished after 01:26:23.972
finished...

created C:\Users\Admin\AppData\Local\Temp\103_1_2024-11-11@02_57_40_5510_02.mkv (7561.97 MB)
starting cleanUpJob for: C:\Users\Admin\AppData\Local\Temp\encodingTempSynthSkript_2024-11-11@02_57_40_5510_0.vpy
delete C:\Users\Admin\AppData\Local\Temp\encodingTempSynthSkript_2024-11-11@02_57_40_5510_0.vpy
starting cleanUpJob for: C:\Users\Admin\AppData\Local\Temp\103_bestSource_2024-11-11@02_57_40_5510
delete C:\Users\Admin\AppData\Local\Temp\103_bestSource_2024-11-11@02_57_40_5510
renamed C:\Users\Admin\AppData\Local\Temp\103_1_2024-11-11@02_57_40_5510_02.mkv to G:\DB KAI part A et B\005-PartA\Davinci Resolve - CC\Hybrid filtred finales\103.mkv


I really do nothing special. Remember that as soon as I load the video into Hybrid, when I want to exit, it tells me that the color matrix is set to GBR, even after a fresh installation, look :

Video


Also, I just saw that the Hybrid video lacks "Color primaries" and "transfer characteristics" in mediainfo, idk why :

[Image: mediainfo.jpg]


Also the Hybrid debug output just in case HybridDebugOutput.zip

I'm still going to uninstall everything and install the last dev to see if there are any changes.
Update : 
  • Hybrid_dev_2024.11.10 won't launch after installation, even after restarting PC, running admin or not, or disabling Antivirus software : it loads into the  task manager for one second then immediately disappear.
  • Hybrid_dev_2024.09.15 and Hybrid_dev_2024.09.26, the previous versions I used before updating to November 5 dev, have no color issue at all on the encoded video (0% difference in diffimg as well). All 3 Color primaries, Transfer characteristics and Matrix coefficients info are present in mediainfo and set to BT.709. And when I load the video and want to exit Hybrid (with all warning boxes checked) I don't have the color matrix changed to GBR issue (only a message about the container set to mkv instead of mp4), so it appears to be a new issue on last versions.


RE: Color change on encoding - Selur - 11.11.2024

Quote: Here is the debug log of one of the problematic videos :
That is not the debug output.

Looking at the vapoursynth script and the encoding call:
"C:\Program Files\Hybrid\64bit\Vapoursynth\vspipe.exe" "C:\Users\Admin\AppData\Local\Temp\encodingTempSynthSkript_2024-11-11@02_58_55_091413_0.vpy" - -c y4m | "C:\Program Files\Hybrid\64bit\ffmpeg.exe" -y -noautorotate -nostdin -threads 8 -f yuv4mpegpipe -i - -an -sn -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range tv -pix_fmt yuv420p -fps_mode passthrough -vcodec ffv1 -coder 0 -context 0 -g 1 -level 1 -metadata encoding_tool="Hybrid 2024.11.05.1" "C:\Users\Admin\AppData\Local\Temp\116_14_2024-11-11@02_58_55_091413_02.mkv"
and the used script:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
import ctypes
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("C:/Program Files/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll")
# loading plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/GrainFilter/RemoveGrain/RemoveGrainVS.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/CTMF/CTMF.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/TTempSmooth/TTempSmooth.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/DCTFilter.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DeblockFilter/Deblock/Deblock.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/BestSource/BestSource.dll")
# Import scripts
import psharpen
import havsfunc
import validate
# Source: 'G:\DB KAI part A et B\005-PartA\Davinci Resolve - CC\116.mkv'
# Current color space: YUV420P8, bit depth: 8, resolution: 1920x1080, frame rate: 23.976fps, scanorder: progressive, yuv luminance scale: limited, matrix: 709, transfer: bt.709, primaries: bt.709, format: FFV1
# Loading G:\DB KAI part A et B\005-PartA\Davinci Resolve - CC\116.mkv using BestSource)
clip = core.bs.VideoSource(source="G:/DB KAI part A et B/005-PartA/Davinci Resolve - CC/116.mkv", cachepath="C:/Users/Admin/AppData/Local/Temp/116_bestSource_2024-11-11@02_58_55_091413", track=0)
frame = clip.get_frame(0)
# setting color matrix to 709.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT709)
# setting color transfer (vs.TRANSFER_BT709), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT709)
# setting color primaries info (to vs.PRIMARIES_BT709), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT709)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED)
# making sure frame rate is set to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive
# denoising using MCTemporalDenoise
clip = havsfunc.MCTemporalDenoise(i=clip, settings="low", thSAD=400, thSAD2=400, thSCD1=400, thSCD2=100, truemotion=False, MVglobal=True, pel=4, pelsearch=5, search=5, searchparam=5, MVsharp=2, DCT=0, ncpu=1)
# sharpening using PSharpen
clip = psharpen.psharpen(clip=clip, strength=16.00)
# applying FineDeHalo to remove halos
clip = havsfunc.FineDehalo(clip, rx=2.50, ry=2.50, darkstr=0.00, brightstr=1.10)
# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# output
clip.set_output()
the whole encoding chain does not contain any rgb<>yuv conversion, so during the filtering color matrix is not used.
So any changes in color are either due to one of the filters you used, this you should be able to see if you use the compare view to compare input and filtered input or it's caused by some tagging.

Quote:I just saw that the Hybrid video lacks "Color primaries" and "transfer characteristics" in mediainfo, idk why
Strange thing is that when I reencoded the clip (with the call I posted) I get:
Video
ID                                       : 1
Format                                   : FFV1
Format version                           : Version 1
Codec ID                                 : V_MS/VFW/FOURCC / FFV1
Duration                                 : 6 min 28 s
Bit rate mode                            : Variable
Bit rate                                 : 119 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Compression mode                         : Lossless
Bits/(Pixel*Frame)                       : 2.394
Stream size                              : 5.38 GiB (98%)
Writing library                          : Lavc61.24.100 ffv1
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
coder_type                               : Golomb Rice

Quote: I'm still going to uninstall everything and install the last dev to see if there are any changes.
Please do, make sure to remove your settings on uninstall.
Also, try not using BestSource.


Cu Selur

RE: Color change on encoding - Selur - 11.11.2024

Quote:Hybrid_dev_2024.11.10 won't launch after installation, even after restarting PC, running admin or not, or disabling Antivirus software : it loads into the task manager for one second then immediately disappear.
I'll look into it.

RE: Color change on encoding - Lipomo - 11.11.2024

I didn't used any filters on my tests, just encoding the video from FFV1 to FFV1 without anything applied.

Also I don't know if you this part of my update message :

Quote:
  • Hybrid_dev_2024.09.15 and Hybrid_dev_2024.09.26, the previous versions I used before updating to November 5 dev, have no color issue at all on the encoded video (0% difference in diffimg as well). All 3 Color primaries, Transfer characteristics and Matrix coefficients info are present in mediainfo and set to BT.709. And when I load the video and want to exit Hybrid (with all warning boxes checked) I don't have the color matrix changed to GBR issue (only a message about the container set to mkv instead of mp4), so it appears to be a new issue on last versions.

I only changed Hybrid version, all other things being equal, it'd make sense that the issue comes from last Hybrid versions.

RE: Color change on encoding - Selur - 11.11.2024

atm. I'm trying to reproduce "Hybrid_dev_2024.11.10 won't launch after installation"
=> found it (deletion order of not installed stuff was wrong)

RE: Color change on encoding - Lipomo - 11.11.2024

Alright

RE: Color change on encoding - Selur - 11.11.2024

Starting with a resetted Hybrid (reset Defaults), enabling all Warnings, loading your source, setting 'FFV1' as encoder, creating a job with mkv container and then closing Hybrid.
I, too, see:
....
<HybridData name="mkvColorMatrix" value="BT709"/>
and
<HybridData name="mkvColorMatrix" value="GBR"/>
differ.

<HybridData name="mkvColorRange" value="Broadcast Range"/>
and
<HybridData name="mkvColorRange" value="Unspecified"/>
differ.

<HybridData name="mkvColorTransferCharacteristics" value="ITU-R BT.709"/>
and
<HybridData name="mkvColorTransferCharacteristics" value="Reserved"/>
differ.

<HybridData name="mkvColourPrimaries" value="ITU-R BT.709"/>
and
<HybridData name="mkvColourPrimaries" value="Reserved"/>
differ.
...
This is correct, since these are setting that changed.
For example, the default:
"<HybridData name="mkvColorMatrix" value="GBR"/>"
changed to
"<HybridData name="mkvColorMatrix" value="BT709"/>"

You told Hybrid it should report any changes, and thus it does.
If you don't want this, do not enable all warnings.
This has nothing to do with color changes.

Using 2024.11.11, loading your file, setting FFV1 as encoder I get:
Input #0, matroska,webm, from 'g:\Output\250frames.mkv':
  Metadata:
    ENCODING_GUI    : Hybrid 2024.11.10.1
    ENCODER         : Lavf61.9.100
  Duration: 00:00:10.43, start: 0.000000, bitrate: 156854 kb/s
  Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv420p(tv, bt709, progressive), 1920x1080, SAR 1:1 DAR 16:9, 23.98 fps, 23.98 tbr, 1k tbn (default) (forced)
      Metadata:
        ENCODER         : Lavc61.24.100 ffv1
        BPS             : 111312176
        NUMBER_OF_FRAMES: 9303
        NUMBER_OF_BYTES : 5398821425
        _STATISTICS_WRITING_APP: mkvmerge v88.0 ('All I Know') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2024-11-11 04:21:23
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
        DURATION        : 00:00:10.427000000
Strangely, when I look at the file with mediainfo it reports a wrong length, frame rate and size values for the video.
Video
ID                                       : 1
Format                                   : FFV1
Format version                           : Version 1
Codec ID                                 : V_MS/VFW/FOURCC / FFV1
Duration                                 : 10 s 427 ms
Bit rate mode                            : Variable
Bit rate                                 : 111 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 892.203 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Compression mode                         : Lossless
Bits/(Pixel*Frame)                       : 0.060
Stream size                              : 5.03 GiB
Writing library                          : Lavc61.24.100 ffv1
Default                                  : Yes
Forced                                   : Yes
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
coder_type                               : Golomb Rice
You do see different color right? You are not just relying on what mediainfo reported, correct?

=> Does the new 2024.11.11 start for you?


Cu Selur

RE: Color change on encoding - Lipomo - 11.11.2024

I'm not complaining that Hybrid is notifying me changes in settings when closing, I'm just surprised why for these sources only it changes to GBR,  which seems to be the origin of the problem. 

I do see color changes on newer hybrid version only, which look similar to a 601 to 709 conversion, or the other way around.

I will try the 2024.11.11 version if you uploaded a new one.

RE: Color change on encoding - Selur - 11.11.2024

It does not change to GBR it changes from GBR. That popup shows 'new value' followed by 'old value', not the other way around, like you seem to assume.

This is not the cause of the problem.
These settings are only used if:
a. "Config->MKV tagging->VUI->Matroska advanced video signaling" is enabled
and
b. mkvtoolnix is used for mkv multiplexing.

Quote:I will try the 2024.11.11 version if you uploaded a new one.
is in the GoogleDrive share.

Cu Selur