[BUG] Hybrid/FFmpeg ProRes 4444 12 bit some problem

[shijan]

It seems Hybrid/FFmpeg now finally able to read ProRes 4444 as true 12 bit and this is really great https://trac.ffmpeg.org/ticket/8054
But i guess due these changes there are some new bugs:
Vapoursynth->Coloir->Levels and Limiter.
If i use ProRes444 12bit and check UseRGB, i got black screen preview. No problem with ProRes422 10bit.
I guess problem may be related to all plug-ins that have "UseRGB" option when used with ProRes 12bit and newest FFmpeg development version.

And another problem - If source is ProRes 444 12 bit, ProRes 444 is still exported only as 10 bit instead of 12 bit. Or most likely processed as 10 bit and placed inside 12 bit container on render. Is this is an FFmpeg limitation or some bug in Hybrid?

ProRes 422 10bit:

# making sure input color matrix is set as 709
clip = core.resize.Point(clip, matrix_in_s="709",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting color space from YUV422P10 to RGB30 for vsLevels
clip = core.resize.Bicubic(clip=clip, format=vs.RGB30, matrix_in_s="709", range_s="limited")
# Color Adjustment
clip = core.std.Levels(clip=clip, min_in=396, max_in=488, min_out=64, max_out=940)
# adjusting output color from: RGB30 to YUV444P16 for ProResModel (i444@10)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="709", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

ProRes444 12bit:

# making sure input color matrix is set as 709
clip = core.resize.Point(clip, matrix_in_s="709",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting color space from YUV444P16 to RGB48 for vsLevels
clip = core.resize.Bicubic(clip=clip, format=vs.RGB48, matrix_in_s="709", range_s="limited")
# Color Adjustment
clip = core.std.Levels(clip=clip, min_in=1584, max_in=1952, min_out=256, max_out=3760)
# adjusting output color from: RGB48 to YUV444P16 for ProResModel (i444@12)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="709", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

[Selur]

If i use ProRes444 12bit and check UseRGB, i got black screen preview. No problem with ProRes422 10bit.
I guess problem may be related to all plug-ins that have "UseRGB" option when used with ProRes 12bit and newest FFmpeg development version.

It's probably either a problem with the:
a. the YUV 4:4:4 16bit <> RGB48 conversion
or
b. a problem with the filter to use RGB48

Since:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# source: 'G:\TestClips&Co\files\ProRes\Test Patterns Resolve 4444 12-bit.mov'
# current color space: YUV444P16, bit depth: 12, resolution: 720x576, fps: 25, color matrix: 709, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\ProRes\Test Patterns Resolve 4444 12-bit.mov using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="G:/TestClips&Co/files/ProRes/Test Patterns Resolve 4444 12-bit.mov")
# making sure input color matrix is set as 709
clip = core.resize.Point(clip, matrix_in_s="709",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting color space from YUV444P16 to RGB48 for vsLevels
clip = core.resize.Bicubic(clip=clip, format=vs.RGB48, matrix_in_s="709", range_s="limited")
# Output
clip.set_output()

properly outputs RGB48, seems like it's a limitation or bug in levels.
-> I asked the Vapoursynth developers over at doom9, to figure out whether it's a bug/limitation in the filter or whether Hybrid is doing something wrong.

If source is ProRes 444 12 bit, ProRes 444 is still exported only as 10 bit instead of 12 bit. Or maybe processed as 10 bit and placed inside 12 bit container on render. Is this is an FFmpeg limitation or some bug in Hybrid?

No details there. The Vapoursynth scripts you showed both output YUV444P16.

Encoding without Vapoursynth:

ffmpeg -y -loglevel fatal -noautorotate -nostdin -threads 8 -ignore_editlist true -i "G:\TestClips&Co\files\ProRes\Test Patterns Resolve 4444 12-bit.mov" -map 0:0 -an -sn -vf zscale=rangein=tv:range=tv -pix_fmt yuv420p12le -strict -1 -vsync 0 -f yuv4mpegpipe - | x265 --input - --output-depth 12 --y4m --profile main444-12 --limit-modes --no-early-skip --no-open-gop --opt-ref-list-length-pps --crf 18.00 --opt-qp-pps --cbqpoffs -2 --crqpoffs -2 --limit-refs 0 --ssim-rd --psy-rd 2.50 --rdoq-level 2 --psy-rdoq 10.00 --aq-mode 0 --deblock=-1:-1 --limit-sao --no-repeat-headers --range limited --colormatrix bt709 --output "E:\Temp\2021-07-26@05_29_19_2410_01.265"

seems to feed 12bit to the encoder.

Looking at the output MediaInfo reports:

Writing library                          : x265 3.5+10-82786fccc:[Windows][GCC 10.3.0][64 bit] 10bit

looking at the output or 'x265 --help', it properly reports:

x265 [info]: HEVC encoder version 3.5+10-82786fccc
x265 [info]: build info [Windows][GCC 10.3.0][64 bit] 8bit+10bit+12bit

-> Strange, may be some one over at doom9 knows what's happening there.

Cu Selur

[shijan]

The most interesting thing - if i import ProRes 444 12 bit test gradient file inside Hybrid and apply Levels - it is visible in preview as true 12 bit. Smooth gradient no 10bit banding.

But if i import ProRes 444 12 bit test gradient, render it without any filters to same ProRes444 and  import rendered file to Hybrid - it became visually look as 10 bit, but seems like placed inside 12 bit container because produce black preview when UseRGB enabled (same as original ProRes 444 12 bit file).

[shijan]

By the way it was another ticket for ProRes 12 bit reader support. Maybe info from that ticket may help somehow https://trac.ffmpeg.org/ticket/7163

---

One more interesting test result. If i render with Hybrid to ProRes 444 MKV (instead of original MOV container) and put that rendered MKV file back to Hybrid - i got black screen when apply Levels even if UseRGB is unchecked.

Also if i use ProRes444 file, apply Levels with any settings and render to ProRes 444 - i got MOV file that is simply black.

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# source: '/Users/aaa/Desktop/resolve prores444-hybrid.mkv'
# current color space: YUV444P16, bit depth: 12, resolution: 1440x1080, fps: 25, color matrix: 709, yuv luminance scale: limited, scanorder: progressive
# Loading /Users/aaa/Desktop/resolve prores444-hybrid.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="/Users/aaa/Desktop/resolve prores444-hybrid.mkv", format="YUV444P16", cache=0, prefer_hw=0)
# making sure input color matrix is set as 709
clip = core.resize.Point(clip, matrix_in_s="709",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# Color Adjustment
clip = core.std.Levels(clip=clip, min_in=1584, max_in=3552, min_out=256, max_out=3760, planes=[0])
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

[Selur]

Problem is with the scaling of the numbers.
Hybrid scales for 12bit, but needs to scale for 16bit.
-> will look at it after work.

Cu Selur

[shijan]

Ok, Hybrid_dev_mac_2021072601.zip works well. No more black screen.
Due native ProRes 12 bit input support and processing at higher bit depths quality became slightly better now.
Too bad FFmpeg still can't render to 12 bit, but i guess it is a problem of FFmpeg developers.

Here are some test results with applied Levels:

Original Resolve ProRes 444 12 bit export:


Same file Exported from Hybrid to ProRes 444 12 bit (current dev version of Hybrid and FFmpeg):


Same file but Exported from legacy Hybrid to ProRes 444 12 bit (Hybrid 2020 and some earlier version of FFmpeg):


And for reference here is Original Resolve ProRes 422 10 bit export:

[Selur]

To know whether this is a handling problem or an encoder problem:
What does the Hybrid output look like when you use x264 or x265 lossless output instead of ProRes (where due to the lossy nature losses are to be expected)?

Cu Selur

[shijan]

Did you mean what is Preview looks like or what is VS script looks?
As you may remember, i tested x264 x265 with same gradients long time ago and confirmed that Hybrid can render x265 to true 12 bit.

---

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# source: '/Volumes/aaa/Bit Depth Tests/hybrid x265 L4.1 RF1 Main-12 High 12bit.mp4'
# current color space: YUV420P16, bit depth: 12, resolution: 1440x1080, fps: 25, color matrix: 709, yuv luminance scale: limited, scanorder: progressive
# Loading /Volumes/aaa/Bit Depth Tests/hybrid x265 L4.1 RF1 Main-12 High 12bit.mp4 using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="/Volumes/aaa/Bit Depth Tests/hybrid x265 L4.1 RF1 Main-12 High 12bit.mp4")
# making sure input color matrix is set as 709
clip = core.resize.Point(clip, matrix_in_s="709",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting color space from YUV420P16 to RGB48 for vsLevels
clip = core.resize.Bicubic(clip=clip, format=vs.RGB48, matrix_in_s="709", range_s="limited")
# Color Adjustment using Levels on RGB48 (16 bit)
clip = core.std.Levels(clip=clip, min_in=25344, max_in=28416, min_out=4096, max_out=60160)
# adjusting output color from: RGB48 to YUV444P16 for ProResModel (i444@12)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="709", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

[Selur]

Question simply is:
a. is it a Hybrid problem
b. is a ProRes recompression problem, since I'm not sure this isn't a compression loss issue.
c. is it a ProRes implementation in FFmpeg problem.
if it looks fine with lossless x264/x265 it isn't a. in general, which would leave b. or c.

Cu Selur

[shijan]

Andrew Kolakowski on BMD forum respond that it is a ProRes implementation in FFmpeg problem:
https://forum.blackmagicdesign.com/viewt...23#p777446