I open a mpg muxed DVD (no re-encoding), and on the x264 tab, the colormatrix is correctly set at bt470bg. (There is no color space tagging in the metadata.)
It's a PAL DVD flagged as top field first, but is mostly progressive with some interlaced.
On the (De-)Interlace/Telecine tab, as soon as I tick the "Overwrite input type scan to:" checkbox, the colormatrix option (on the x264 tab) is set to bt709. It will stay there no matter what I set or do in Hybrid (un-tick Overwrite input scan, turn on AviSynth32 ColorMatrix filter, etc.) unless the file is reloaded.
If the ColorMatrix Avisynth32 filter is enabled (709 to 601) *before* the input file is loaded, then the colormatrix option on x264 will stay correct at bt470bg, even if Overwrite input type is enabled. As soon as ColorMatrix AviSynth32 filter is disabled, if the Overwrite input type is enabled, colormatrix on x264 tab will go to bt709 and stay there until file is reloaded.
I believe you should be able to reproduce this with any SD input file that is 720x480 or 720x576, that is untagged for color space. Can be worked around by setting Color matrix option on VUI tab via x264 tab for Advanced mode.
It's a PAL DVD flagged as top field first, but is mostly progressive with some interlaced.
On the (De-)Interlace/Telecine tab, as soon as I tick the "Overwrite input type scan to:" checkbox, the colormatrix option (on the x264 tab) is set to bt709. It will stay there no matter what I set or do in Hybrid (un-tick Overwrite input scan, turn on AviSynth32 ColorMatrix filter, etc.) unless the file is reloaded.
If the ColorMatrix Avisynth32 filter is enabled (709 to 601) *before* the input file is loaded, then the colormatrix option on x264 will stay correct at bt470bg, even if Overwrite input type is enabled. As soon as ColorMatrix AviSynth32 filter is disabled, if the Overwrite input type is enabled, colormatrix on x264 tab will go to bt709 and stay there until file is reloaded.
I believe you should be able to reproduce this with any SD input file that is 720x480 or 720x576, that is untagged for color space. Can be worked around by setting Color matrix option on VUI tab via x264 tab for Advanced mode.