Selur's Little Message Board
[BUG] Interlace Store Method Changing Upon Conversion - Printable Version

+- Selur's Little Message Board (https://forum.selur.net)
+-- Forum: Hybrid - Support (https://forum.selur.net/forum-1.html)
+--- Forum: Problems & Questions (https://forum.selur.net/forum-3.html)
+--- Thread: [BUG] Interlace Store Method Changing Upon Conversion (/thread-2847.html)

Pages: 1 2 3

Interlace Store Method Changing Upon Conversion - Analog - 06.08.2022

Hi Selur,

I'm back to bug you again!

I'm having a big problem with dev release 2022.07.18.1 where the Interlace "Scan type" & "Scan type, store method" is changing upon processing thru Hybrid. This causes some big problems with deinterlacing (although it doesn't seem to cause issues with the first conversion when leaving as interlace which is why I didn't notice it until now).

Please see the zip file here: https://drive.google.com/file/d/1l1Od1N8gNo_xIayIcw397rKD6kWE5sx2/view?usp=sharing

Input.mov is the input file while Output.mkv is the output file. Input.mov has an store method of "Interleaved fields" with a scan type of "Bottom Field First" but the output.mkv has a store type of "Separated fields (2 fields per block)" with a scan type of "Top Field First." Using the Test profile and importing Input.mov will result in Output.mkv.

My old conversions from the April dev version of Hybrid show that the field order header parameter should be "14" but Output.mkv has a parameter value of "1."

I am luckily able to recover my interlaced conversions with MKVToolNix but I can't produce valid deinterlaced video with this dev release. Any assistance with this would be great as this is a show stopper for me!

Thanks!

RE: Interlace Store Method Changing Upon Conversion - Selur - 06.08.2022

mkv supports, the following field orders:
  • 0: progressive
  • 1: interlaced with top field displayed first and top field stored first
  • 2: undetermined field order
  • 6: interlaced with bottom field displayed first and bottom field stored first
  • 9: interlaced with bottom field displayed first and top field stored first
  • 14: interlaced with top field displayed first and bottom field stored first
but Hybrid only supports using 0, 1, 6.

I never had a use case where I needed to save interlaced content as interleaved.
Interlaced content can be coded in different manners:
- Interleaved fields : line 0, 1, 2, 3 ... 476, 477, 478, 479 (in 1 block), similar to how progressive content is coded
- Separated fields: line 0, 2 ... 476, 478, 1, 3 ... 477, 479 (2 fields in 1 block or each field in its own block)
see: https://www.mir.com/DMG/interl.html
Nor have I ever encountered that anyone used it for formats that are newer than MPEG-2, which is why Hybrid doesn't use it.

If I wanted to add supporting the output to interleaved content:
  • I would need to know whether I can create such content using FFmpeg and MP4Box or does only mkvmerge support this? Which containers suppor this.
  • I would have to decide how to add this to the gui. Atm. the best way would be to add some 'store interlaced output' combobox with the options:
    • same as source (prefer separated): will use interleaved fields if the input used interleaved fields and otherwise separated fields.
    • same as source (prefer interleaced): will use separated fields if the input used separated fields and otherwise interleaved fields.
    • separated fields: interlaced output will always be saved with separated fields (default)
    • interleaved fields: interlaced output will always be saved with interleaved fields.
  • I would have to rewrite the analysis and muxing code in Hybrid to
    • detect (and remember) whether the input used interleaved or separated fields.
    • respect the 'store interlaced output' selection.

Main issue is the first point: I don't know how saving interlaced content with separated fields could be done with anything other than mkvmerge.
I do not plan to only add this as an advanced mkvmerge option.

I adjusted Hybrid to inform the user on interlaced input where the output also is interlaced that the store type will be changed to separated fields.
Quote:Notice: Input used interleaved fields to store interlacing while output uses separate fields.
Hybrid does not support saving interlaced content with interleaved fields.
will be added to the normal user log.

So atm. you probably either have to:
  • use a different tool,
  • or change the store type with mkvmerge by remuxing.
  • provide me with the info how to save interlaced content using ffmpeg and mp4box if you want me to support this in Hybrid.

Cu Selur

Ps.: the more I have to deal with interlaced output the more I think about dropping the whole thing since it seems to pointless. *gig*

RE: Interlace Store Method Changing Upon Conversion - Analog - 06.08.2022

Hi Selur,

Thanks for the extensive reply.

I'm going to reply quickly because its late/early here in the USA.

Quote:Main issue is the first point: I don't know how saving interlaced content with separated fields could be done with anything other than mkvmerge.

I do not plan to only add this as an advanced mkvmerge option.

I think there might be some confusion here (maybe on my part). The video output that is being generated by Hybrid is still interleaved, not separated yet it is being tagged as separated. Then, if I used the output file (not the input file) for deinterlacing, it is processed wrong because it is tagged wrong. The progressive output file at that point is unusable. I have no need and see no benefit to have any level of UI/processing control over how the interlace is stored since it's visually identical in the end.

Since, obviously, the interlace storage method is maintained through Hybrid processing and all you're setting is bff/tff, couldn't the tag "group" just be chosen from the source file field storage order and appended to the final tags? This would be your last bullet point to implement, I think. I believe this is how Hybrid treated interlace in general in the past and it worked beautifully. Once again, FFMpeg seems to just "go with the flow" when it comes to the source file storage format. Only the tag needs to be correct...

Does this thread help at all? https://forum.videohelp.com/threads/402926-Best-archival-codec-in-ffmpeg-for-preservation-of-interlaced-fields#post2629750

Quote:Nor have I ever encountered that anyone used it for formats that are newer than MPEG-2, which is why Hybrid doesn't use it.
Unfortunately, this is the way Blackmagic chose to handle interlaced captures... It's quite common in the archival scene, not in any consumer formats.

Alternatively, do you have the your old dev build from April that you had supplied me. That worked well for me for deinterlacing and I didn't back it up...

Quote:Ps.: the more I have to deal with interlaced output the more I think about dropping the whole thing since it seems to pointless. *gig*

Please don't! This is one of the only easy-to-use tools that works well with interlace and we (as in the whole world) have 40+ years of analog video sources to archive!

RE: Interlace Store Method Changing Upon Conversion - Selur - 06.08.2022

Quote:I think there might be some confusion here (maybe on my part). The video output that is being generated by Hybrid is still interleaved, not separated yet it is being tagged as separated
Internally mkv always saves interlaced content separated by field (in the container structure) and the actual way the content is saved in the container does differ with the field-order selection on muxing. It's not just a tag.
Quote: couldn't the tag just be maintained from the source file and appended to the final tags?
It's not just a tag, also like I wrote I don't know how to tell ffmpeg and mp4box how to tell them to save and signal interlaced contet as interleaved fields.

Quote:Alternatively, do you have the your old dev build from April that you had supplied me
ARGH,... do !not! post links I send via PM (edited your post) or I will delete your account.
iirc I changed the way Hybrid handles the muxing of interlaced content to mkv due to changes in mkv since there were some changes a while ago in mkv and this causes Hybrid to not properly save interlaced content.
This is the way it is now and how it will stay unless I would add explicit support for interlaced storage hangling like I sketched out in my previous post, since this is the way that mkvmerge properly saved normal interlaced content.

Quote:have 40+ years of analog video sources to archive!
Personally I would bob the content and save it as progressive and not interlaced as more and more newer formats drop support for interlaced content altogether.

Cu Selur

RE: Interlace Store Method Changing Upon Conversion - Analog - 06.08.2022

Quote: Wrote:Internally mkv always saves interlaced content separated by field (in the container structure) and the actual way the content is saved in the container does differ with the field-order selection on muxing. It's not just a tag.
I'm still confused on this one. I get that it can be stored differently. That makes sense.

But FFmpeg is quite clearly just "going with the flow" and processing the file/outputting using the storage order from the source file, but then Hybrid is overwriting that when writing the tags because it doesn't know the original storage order. Otherwise, the output file would have actually changed storage order, right? The output file is still the original storage order just tagged wrong.


Can't Hybrid read the field order tag from the input file and then make a decision like "if input field order is is 1 or 9 && input/processing is bff, write field order tag as 9, else 1. If input field order is is 6 or 14 && input/processing is tff, write field order tag as 14, else 6"?


Quote:ARGH,... do !not! post links I send via PM (edited your post) or I will delete your account.


Sorry, figured dead links were permissible.

Is there no way to get older dev versions of Hybrid?


Quote:Personally I would bob the content and save it as progressive and not interlaced as more and more newer formats drop support for interlaced content altogether.

People were saying that way more than 12 years ago before QTGMC existed...yet algorithms seem to keep getting better! Smile (QTGMC just turned 12 a couple days ago apparently!)

RE: Interlace Store Method Changing Upon Conversion - Selur - 06.08.2022

I just noticed there is a bug in Hybrid. The way it saves interlaced FFV1!

it reports in the FFV1-tab:
-vcodec ffv1 -coder 0 -context 0 -g 1 -level 1

-vcodec ffv1 -ovc lavc -lavcopts vcodec=ffv1:coder=0:context=0:version=1 -flags +ildct+ilme -top 0 -vf setfield=bff
but the problem is it only uses the first line:
ffmpeg -y -noautorotate -nostdin -threads 8 -ignore_editlist true -i "C:\Users\Selur\Desktop\input.mov" -map 0:0 -an -sn -vf zscale=rangein=tv:range=tv -pix_fmt yuv422p -vsync 0 -vcodec ffv1 -coder 0 -context 0 -g 1 -level 1 -metadata encoding_tool="Hybrid 2022.08.06.1" "E:\Output\input_2022-08-06@11_40_29_4710_01.mkv"
thus it doesn't really tell ffmpeg to encode the content as interlaced,...

Have to look into this some more.

-> ah no, 1st one is the ffmpeg call, second one it the mencoder call that would be used for ffv1 encoding, but the ' -flags +ildct+ilme -top 0 -vf setfield=bff' part should be part of the ffmpeg and not the mencoder call.

Quote:Is there no way to get older dev versions of Hybrid?
no

Quote:Can't Hybrid read the field order tag from the input file and then make a decision like "if input field order is is 1 or 9 && input/processing is bff, write field order tag as 9, else 1. If input field order is is 6 or 14 && input/processing is tff, write field order tag as 14, else 6
That leaves out too many options. Like what if the output is interlaced, but the input wasn't? What if the input was separated fields, but the output should be interleaved fields and the other way around. I sketched how it should be done.
Cu Selur

RE: Interlace Store Method Changing Upon Conversion - Selur - 06.08.2022

Okay, I fixed the FFV1 creation and muxing call.
Due to the error the creation call was tagged as interleaved fields while it used separate fields. From what I read FFV1 decoders can handle this since the look at the bit stream itself. So no real issue, especially since this is an intermediate file.

So as intended, Hybrid by default always uses 'separate fields' when muxing interlaced content.

Support for saving 'interleaved fields' instead will only be added if someone provides the needed info on how to do it.
If someone need 'interleaved fields' output an additional remux is needed.

Cu Selur

Ps.: send you a pm with a link to the dev version.

RE: Interlace Store Method Changing Upon Conversion - Analog - 06.08.2022

Hi Selur,

Thanks for finding that bug!!! It looks like that corrected the storage method as it now appears to be true separated fields on the output.

However, it seems to also be tagging the file as tff when it is really bff. When I create an ffv1 output file and then use the output for deinterlacing, the deinterlaced output shows that it was processed with the fields reversed. Simply setting "Overwrite input scan type" to bff on the Filtering>De-Interlace/Telecine tab fixes the deinterlaced output. Hybrid does note that the original input file is bff and that the imported ffv1 file is tff.

It looks like the output field tag is being set to 1 instead of 6... Can this be corrected?

See attached example/log file here: https://drive.google.com/file/d/11wLYi6rqZ8TwYPgz6_KJmlCgWO7z0iYh/view?usp=sharing

Thanks!

RE: Interlace Store Method Changing Upon Conversion - Selur - 06.08.2022

Ah, I see the issue, a typo in the muxing call.
-> will fix

RE: Interlace Store Method Changing Upon Conversion - Analog - 22.08.2022

Hi Selur,

I wasn't able to try the last debug release of Hybrid until just now due to business travel.

I seem to always get a "C:\XXXX\XXX_2022-08-22@02_18_06_1110_04.mkv is too small! (byteSize: 0byte)" error.

Please see the attached debug log and profile here: https://drive.google.com/file/d/1FTc75r9HJoxeAD9Lm53mbbhnboyUbXCE/view?usp=sharing

Please let me know if you need any more information from me.

Thanks!