[HELP] "BT.709 or BT.2020?" But…

[Selur]

had the color noticably darker and green-ish.

might be a tv (16-235) vs. pc (0-255) scale issue.

So, what's the real problem here? Color matrix?

I doubt it's a color matrix issue unless the player doesn't support the matrix or ignores it and always uses the same matrix (or the same depending on the resolution).
-> if you want to change the color matrix:
a. use Color Matrix filter (https://www.ffmpeg.org/ffmpeg-all.html#toc-colormatrix to convert from one to another color matrix)
b. make sure to properly set the vui settings (color matrix, color range, color primaries)

-color_range XXX -colorspace XXX -color_primaries XXX -color_trc XXX

-> read
a. https://trac.ffmpeg.org/wiki/colorspace is also has some examples.
b. https://www.ffmpeg.org/ffmpeg-all.htm for details on the options

Usually ITU-R recommends for:
1. HDR and UHD content BT.2020
2. SDR and FullHD content BT.709
but there's also UHD content with BT.709 and SDR content with BT.601 or other color matrixes, so there is no strict you need to use color matrix xy for resolution A or B. (Side note: when doing HDR to SDR conversions tome mapping is recommended.)
Problem is not all players support all matrices and some always assume BT.2020 requires HDR which it doesn't. SDR content can have a WCG (wide color gamut, see: https://en.wikipedia.org/wiki/Rec._2020) and you can have WCG without HDR (high dynamic range).
iirc. (not totally sure) to take full advantage of BT.2020 10bit or higher is required (it is for HDR), but it should be possible with 8bit.

-> I would use BT.709 for SDR content independent of the resolution.
And recommend to use BT.2020 only for HDR content (independent of the resolution).

Cu Selur