Selur's Little Message Board
Color change on encoding - Printable Version

+- Selur's Little Message Board (https://forum.selur.net)
+-- Forum: Hybrid - Support (https://forum.selur.net/forum-1.html)
+--- Forum: Problems & Questions (https://forum.selur.net/forum-3.html)
+--- Thread: Color change on encoding (/thread-3933.html)

Pages: 1 2 3

RE: Color change on encoding - Selur - 11.11.2024

For me Hybrid then uses the following command line:
ffmpeg -y -noautorotate -nostdin -threads 8 -i "G:\color_change_099.mkv" -map 0:0 -an -sn -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range tv -pix_fmt yuv420p -fps_mode passthrough -vcodec ffv1 -coder 0 -context 0 -g 1 -level 1 -metadata encoding_tool="Hybrid 2024.11.10.1" "J:\tmp\2024-11-11@14_31_30_9810\color_change_099_reencoded_1_2024-11-11@14_31_30_9810_01.mkv"
which looks fine to me.
Comparing those two I used:
# Imports import vapoursynth as vs # getting Vapoursynth core import sys import os core = vs.core # Import scripts folder scriptPath = 'F:/Hybrid/64bit/vsscripts' sys.path.insert(0, os.path.abspath(scriptPath)) # loading plugins core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/libhistogram.dll") core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll") # Import scripts import validate # Source: 'G:\color_change_099.mkv' # Current color space: YUV420P8, bit depth: 8, resolution: 1920x1080, frame rate: 23.976fps, scanorder: progressive, yuv luminance scale: limited, matrix: 709, transfer: bt.709, primaries: bt.709, format: FFV1 # Loading G:\color_change_099.mkv using LWLibavSource clip = core.lsmas.LWLibavSource(source="G:/color_change_099.mkv", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0) clip = core.hist.Classic(clip=clip) reencode = core.lsmas.LWLibavSource(source="G:/color_change_099_reencoded.mkv", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0) reencode = core.hist.Classic(clip=reencode) clip = core.std.Interleave([clip.text.Text("Org"),reencode.text.Text("Reencode")]) # output clip.set_output()
to load the original and the reencode and then compare them and their histograms.

The output is identical. (also tried different histogram, but as expected they are identical)
Sorry, but I can't produce your problem and if I can't produce it, I can't fix it.
My guess is that, either you are doing something different or for some unknown reason whatever you used to compare these chose to convert them differently to rgb.

Cu Selur

RE: Color change on encoding - Lipomo - 11.11.2024

I ran many encodings of these episodes last night, and all have the issue.

Here is the debug log of one of the problematic videos : 

Created skript C:\Users\Admin\AppData\Local\Temp\encodingTempSynthSkript_2024-11-11@02_57_40_5510_0.vpy:--------------------- # Imports import vapoursynth as vs # getting Vapoursynth core import ctypes import sys import os core = vs.core # Import scripts folder scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts' sys.path.insert(0, os.path.abspath(scriptPath)) # Loading Support Files Dllref = ctypes.windll.LoadLibrary("C:/Program Files/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll") # loading plugins core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/GrainFilter/RemoveGrain/RemoveGrainVS.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/CTMF/CTMF.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/fmtconv.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/TTempSmooth/TTempSmooth.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/DCTFilter.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DeblockFilter/Deblock/Deblock.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/libmvtools.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/BestSource/BestSource.dll") # Import scripts import psharpen import havsfunc import validate # Source: 'G:\DB KAI part A et B\005-PartA\Davinci Resolve - CC\103.mkv' # Current color space: YUV420P8, bit depth: 8, resolution: 1920x1080, frame rate: 23.976fps, scanorder: progressive, yuv luminance scale: limited, matrix: 709, transfer: bt.709, primaries: bt.709, format: FFV1 # Loading G:\DB KAI part A et B\005-PartA\Davinci Resolve - CC\103.mkv using BestSource) clip = core.bs.VideoSource(source="G:/DB KAI part A et B/005-PartA/Davinci Resolve - CC/103.mkv", cachepath="C:/Users/Admin/AppData/Local/Temp/103_bestSource_2024-11-11@02_57_40_5510", track=0) frame = clip.get_frame(0) # setting color matrix to 709. clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT709) # setting color transfer (vs.TRANSFER_BT709), if it is not set. if validate.transferIsInvalid(clip):   clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT709) # setting color primaries info (to vs.PRIMARIES_BT709), if it is not set. if validate.primariesIsInvalid(clip):   clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT709) # setting color range to TV (limited) range. clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED) # making sure frame rate is set to 23.976fps clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001) # making sure the detected scan type is set (detected: progressive) clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive # denoising using MCTemporalDenoise clip = havsfunc.MCTemporalDenoise(i=clip, settings="low", thSAD=400, thSAD2=400, thSCD1=400, thSCD2=100, truemotion=False, MVglobal=True, pel=4, pelsearch=5, search=5, searchparam=5, MVsharp=2, DCT=0, ncpu=1) # sharpening using PSharpen clip = psharpen.psharpen(clip=clip, strength=16.00) # applying FineDeHalo to remove halos clip = havsfunc.FineDehalo(clip, rx=2.50, ry=2.50, darkstr=0.00, brightstr=1.10) # set output frame rate to 23.976fps (progressive) clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001) # output clip.set_output() --------------------- ffmpeg started... starting 2024-11-11@02_57_40_5510_02_video@03:03:38.014 - G:\DB KAI part A et B\005-PartA\Davinci Resolve - CC\Hybrid filtred finales\103.mkv "C:\Program Files\Hybrid\64bit\ffmpeg.exe" -y -noautorotate -nostdin -threads 8 -f yuv4mpegpipe -i - -an -sn -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range tv -pix_fmt yuv420p -fps_mode passthrough -vcodec ffv1 -coder 0 -context 0 -g 1 -level 1 -metadata encoding_tool="Hybrid 2024.11.05.1" "C:\Users\Admin\AppData\Local\Temp\103_1_2024-11-11@02_57_40_5510_02.mkv" 2024-11-11@02_57_40_5510_02_video finished after 01:26:23.972 finished... created C:\Users\Admin\AppData\Local\Temp\103_1_2024-11-11@02_57_40_5510_02.mkv (7561.97 MB) starting cleanUpJob for: C:\Users\Admin\AppData\Local\Temp\encodingTempSynthSkript_2024-11-11@02_57_40_5510_0.vpy delete C:\Users\Admin\AppData\Local\Temp\encodingTempSynthSkript_2024-11-11@02_57_40_5510_0.vpy starting cleanUpJob for: C:\Users\Admin\AppData\Local\Temp\103_bestSource_2024-11-11@02_57_40_5510 delete C:\Users\Admin\AppData\Local\Temp\103_bestSource_2024-11-11@02_57_40_5510 renamed C:\Users\Admin\AppData\Local\Temp\103_1_2024-11-11@02_57_40_5510_02.mkv to G:\DB KAI part A et B\005-PartA\Davinci Resolve - CC\Hybrid filtred finales\103.mkv


I really do nothing special. Remember that as soon as I load the video into Hybrid, when I want to exit, it tells me that the color matrix is set to GBR, even after a fresh installation, look :

Video


Also, I just saw that the Hybrid video lacks "Color primaries" and "transfer characteristics" in mediainfo, idk why :

[Image: mediainfo.jpg]


Also the Hybrid debug output just in case HybridDebugOutput.zip

I'm still going to uninstall everything and install the last dev to see if there are any changes.
Update : 
  • Hybrid_dev_2024.11.10 won't launch after installation, even after restarting PC, running admin or not, or disabling Antivirus software : it loads into the  task manager for one second then immediately disappear.
  • Hybrid_dev_2024.09.15 and Hybrid_dev_2024.09.26, the previous versions I used before updating to November 5 dev, have no color issue at all on the encoded video (0% difference in diffimg as well). All 3 Color primaries, Transfer characteristics and Matrix coefficients info are present in mediainfo and set to BT.709. And when I load the video and want to exit Hybrid (with all warning boxes checked) I don't have the color matrix changed to GBR issue (only a message about the container set to mkv instead of mp4), so it appears to be a new issue on last versions.


RE: Color change on encoding - Selur - 11.11.2024

Quote: Here is the debug log of one of the problematic videos :
That is not the debug output.

Looking at the vapoursynth script and the encoding call:
"C:\Program Files\Hybrid\64bit\Vapoursynth\vspipe.exe" "C:\Users\Admin\AppData\Local\Temp\encodingTempSynthSkript_2024-11-11@02_58_55_091413_0.vpy" - -c y4m | "C:\Program Files\Hybrid\64bit\ffmpeg.exe" -y -noautorotate -nostdin -threads 8 -f yuv4mpegpipe -i - -an -sn -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range tv -pix_fmt yuv420p -fps_mode passthrough -vcodec ffv1 -coder 0 -context 0 -g 1 -level 1 -metadata encoding_tool="Hybrid 2024.11.05.1" "C:\Users\Admin\AppData\Local\Temp\116_14_2024-11-11@02_58_55_091413_02.mkv"
and the used script:
# Imports import vapoursynth as vs # getting Vapoursynth core import ctypes import sys import os core = vs.core # Import scripts folder scriptPath = 'C:/Program Files/Hybrid/64bit/vsscripts' sys.path.insert(0, os.path.abspath(scriptPath)) # Loading Support Files Dllref = ctypes.windll.LoadLibrary("C:/Program Files/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll") # loading plugins core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/GrainFilter/RemoveGrain/RemoveGrainVS.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/CTMF/CTMF.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/fmtconv.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/TTempSmooth/TTempSmooth.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DenoiseFilter/NEO_FFT3DFilter/neo-fft3d.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/DCTFilter.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/DeblockFilter/Deblock/Deblock.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/libmvtools.dll") core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/BestSource/BestSource.dll") # Import scripts import psharpen import havsfunc import validate # Source: 'G:\DB KAI part A et B\005-PartA\Davinci Resolve - CC\116.mkv' # Current color space: YUV420P8, bit depth: 8, resolution: 1920x1080, frame rate: 23.976fps, scanorder: progressive, yuv luminance scale: limited, matrix: 709, transfer: bt.709, primaries: bt.709, format: FFV1 # Loading G:\DB KAI part A et B\005-PartA\Davinci Resolve - CC\116.mkv using BestSource) clip = core.bs.VideoSource(source="G:/DB KAI part A et B/005-PartA/Davinci Resolve - CC/116.mkv", cachepath="C:/Users/Admin/AppData/Local/Temp/116_bestSource_2024-11-11@02_58_55_091413", track=0) frame = clip.get_frame(0) # setting color matrix to 709. clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT709) # setting color transfer (vs.TRANSFER_BT709), if it is not set. if validate.transferIsInvalid(clip): clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT709) # setting color primaries info (to vs.PRIMARIES_BT709), if it is not set. if validate.primariesIsInvalid(clip): clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT709) # setting color range to TV (limited) range. clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED) # making sure frame rate is set to 23.976fps clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001) # making sure the detected scan type is set (detected: progressive) clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive # denoising using MCTemporalDenoise clip = havsfunc.MCTemporalDenoise(i=clip, settings="low", thSAD=400, thSAD2=400, thSCD1=400, thSCD2=100, truemotion=False, MVglobal=True, pel=4, pelsearch=5, search=5, searchparam=5, MVsharp=2, DCT=0, ncpu=1) # sharpening using PSharpen clip = psharpen.psharpen(clip=clip, strength=16.00) # applying FineDeHalo to remove halos clip = havsfunc.FineDehalo(clip, rx=2.50, ry=2.50, darkstr=0.00, brightstr=1.10) # set output frame rate to 23.976fps (progressive) clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001) # output clip.set_output()
the whole encoding chain does not contain any rgb<>yuv conversion, so during the filtering color matrix is not used.
So any changes in color are either due to one of the filters you used, this you should be able to see if you use the compare view to compare input and filtered input or it's caused by some tagging.

Quote:I just saw that the Hybrid video lacks "Color primaries" and "transfer characteristics" in mediainfo, idk why
Strange thing is that when I reencoded the clip (with the call I posted) I get:
Video ID : 1 Format : FFV1 Format version : Version 1 Codec ID : V_MS/VFW/FOURCC / FFV1 Duration : 6 min 28 s Bit rate mode : Variable Bit rate : 119 Mb/s Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate mode : Constant Frame rate : 23.976 (24000/1001) FPS Color space : YUV Chroma subsampling : 4:2:0 Bit depth : 8 bits Compression mode : Lossless Bits/(Pixel*Frame) : 2.394 Stream size : 5.38 GiB (98%) Writing library : Lavc61.24.100 ffv1 Default : Yes Forced : No Color range : Limited Color primaries : BT.709 Transfer characteristics : BT.709 Matrix coefficients : BT.709 coder_type : Golomb Rice

Quote: I'm still going to uninstall everything and install the last dev to see if there are any changes.
Please do, make sure to remove your settings on uninstall.
Also, try not using BestSource.


Cu Selur

RE: Color change on encoding - Selur - 11.11.2024

Quote:Hybrid_dev_2024.11.10 won't launch after installation, even after restarting PC, running admin or not, or disabling Antivirus software : it loads into the task manager for one second then immediately disappear.
I'll look into it.

RE: Color change on encoding - Lipomo - 11.11.2024

I didn't used any filters on my tests, just encoding the video from FFV1 to FFV1 without anything applied.

Also I don't know if you this part of my update message :

Quote:
  • Hybrid_dev_2024.09.15 and Hybrid_dev_2024.09.26, the previous versions I used before updating to November 5 dev, have no color issue at all on the encoded video (0% difference in diffimg as well). All 3 Color primaries, Transfer characteristics and Matrix coefficients info are present in mediainfo and set to BT.709. And when I load the video and want to exit Hybrid (with all warning boxes checked) I don't have the color matrix changed to GBR issue (only a message about the container set to mkv instead of mp4), so it appears to be a new issue on last versions.

I only changed Hybrid version, all other things being equal, it'd make sense that the issue comes from last Hybrid versions.

RE: Color change on encoding - Selur - 11.11.2024

atm. I'm trying to reproduce "Hybrid_dev_2024.11.10 won't launch after installation"
=> found it (deletion order of not installed stuff was wrong)

RE: Color change on encoding - Lipomo - 11.11.2024

Alright

RE: Color change on encoding - Selur - 11.11.2024

Starting with a resetted Hybrid (reset Defaults), enabling all Warnings, loading your source, setting 'FFV1' as encoder, creating a job with mkv container and then closing Hybrid.
I, too, see:
.... <HybridData name="mkvColorMatrix" value="BT709"/> and <HybridData name="mkvColorMatrix" value="GBR"/> differ. <HybridData name="mkvColorRange" value="Broadcast Range"/> and <HybridData name="mkvColorRange" value="Unspecified"/> differ. <HybridData name="mkvColorTransferCharacteristics" value="ITU-R BT.709"/> and <HybridData name="mkvColorTransferCharacteristics" value="Reserved"/> differ. <HybridData name="mkvColourPrimaries" value="ITU-R BT.709"/> and <HybridData name="mkvColourPrimaries" value="Reserved"/> differ. ...
This is correct, since these are setting that changed.
For example, the default:
"<HybridData name="mkvColorMatrix" value="GBR"/>"
changed to
"<HybridData name="mkvColorMatrix" value="BT709"/>"

You told Hybrid it should report any changes, and thus it does.
If you don't want this, do not enable all warnings.
This has nothing to do with color changes.

Using 2024.11.11, loading your file, setting FFV1 as encoder I get:
Input #0, matroska,webm, from 'g:\Output\250frames.mkv': Metadata: ENCODING_GUI : Hybrid 2024.11.10.1 ENCODER : Lavf61.9.100 Duration: 00:00:10.43, start: 0.000000, bitrate: 156854 kb/s Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv420p(tv, bt709, progressive), 1920x1080, SAR 1:1 DAR 16:9, 23.98 fps, 23.98 tbr, 1k tbn (default) (forced) Metadata: ENCODER : Lavc61.24.100 ffv1 BPS : 111312176 NUMBER_OF_FRAMES: 9303 NUMBER_OF_BYTES : 5398821425 _STATISTICS_WRITING_APP: mkvmerge v88.0 ('All I Know') 64-bit _STATISTICS_WRITING_DATE_UTC: 2024-11-11 04:21:23 _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES DURATION : 00:00:10.427000000
Strangely, when I look at the file with mediainfo it reports a wrong length, frame rate and size values for the video.
Video ID : 1 Format : FFV1 Format version : Version 1 Codec ID : V_MS/VFW/FOURCC / FFV1 Duration : 10 s 427 ms Bit rate mode : Variable Bit rate : 111 Mb/s Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate mode : Variable Frame rate : 892.203 FPS Color space : YUV Chroma subsampling : 4:2:0 Bit depth : 8 bits Compression mode : Lossless Bits/(Pixel*Frame) : 0.060 Stream size : 5.03 GiB Writing library : Lavc61.24.100 ffv1 Default : Yes Forced : Yes Color range : Limited Color primaries : BT.709 Transfer characteristics : BT.709 Matrix coefficients : BT.709 coder_type : Golomb Rice
You do see different color right? You are not just relying on what mediainfo reported, correct?

=> Does the new 2024.11.11 start for you?


Cu Selur

RE: Color change on encoding - Lipomo - 11.11.2024

I'm not complaining that Hybrid is notifying me changes in settings when closing, I'm just surprised why for these sources only it changes to GBR,  which seems to be the origin of the problem. 

I do see color changes on newer hybrid version only, which look similar to a 601 to 709 conversion, or the other way around.

I will try the 2024.11.11 version if you uploaded a new one.

RE: Color change on encoding - Selur - 11.11.2024

It does not change to GBR it changes from GBR. That popup shows 'new value' followed by 'old value', not the other way around, like you seem to assume.

This is not the cause of the problem.
These settings are only used if:
a. "Config->MKV tagging->VUI->Matroska advanced video signaling" is enabled
and
b. mkvtoolnix is used for mkv multiplexing.

Quote:I will try the 2024.11.11 version if you uploaded a new one.
is in the GoogleDrive share.

Cu Selur