Selur's Little Message Board

As i've been doing a lot of test encodes lately, which mostly just consists of tediously copy-pasting Encoding settings and filesizes to text files for comparison in WinMerge, i've come up with some ideas that would make this sort of thing a whole lot faster...

1. Support adding metadata like Filesize, Bitrate, PSNR, SSIM & Encoding settings to automatically generated filenames.
   • Since adding Encoding settings could easily result in a filename that's way too long, also allow blacklisting of user specified / redundant text strings like so:
   Blacklist:
   Code:

   cpuid=*; frame-threads=*; numa-pools=*; no-pmode; no-pme; psnr; ssim; log-level=2; input-csp=1; input-res=1920x1080; interlace=0; total-frames=0; level-idc=0; high-tier=1; uhd-bd=0; ref=6; no-allow-non-conformance; no-repeat-headers; annexb; no-aud; no-hrd; info; hash=0
   • With '*' being a wildcard and ';' acting as separator. Thus user would be able to copy-paste their baseline settings here and have the filename only include the changed settings.
   
   
2. When specified, use automatically generated filenames also for intermediate Video, Audio & Subtitle tracks, each with their own Filesize(in bytes), bitrate(in Kbps), etc. metadata; renaming can be done once encoding is finished.
   
   
3. The two suggestions above would make this almost obsolete, but allowing for serialization of filenames, where for example: 'test#{2}' would become 'test#01' unless it already exists, in which case it'll become 'test#02' as opposed to just replacing the older file. This would help avoid accidental overwrites. I haven't really given much thought to whether '{#}' would be a good markup (there could be more commonly used alternative), but the general idea is to allow for padding with zeros to a specified length.
   
   
4. To keep things cleaner, I've been using a simple trick to do my encodings inside a subfolder of the source file's directory by setting 'enc\' as prefix for the filename. However .chp & .xml files are still being created inside the source file's folder. Perhaps there is a reason why these need to be there during encoding, but moving or deleting them automatically after encoding would be helpful, especially since a new one gets created every time the file is encoded, thereby making the folder slower to navigate and easier for user to accidentally delete something they shouldn't. EDIT: This is about adding full support for dynamic destination directories, not about improving or fixing an accidental 'feature'.
   

PS. I'm gonna see if i can create a poll on this new forum system to see whether these ideas have any traction within the userbase.

My two cents:
Regarding 1.:
This request should go to the x265 developers not me.

Regarding 2:
Not going to happen. Intermediate files did at one point contain lots of meta data which causes lots of problems on Windows due to lack of UTF-8 support through a bunch of tools and some Windows combinations. To avoid those problems Hybrid now should use generated names based on the job ID and this is the way it will stay.

Regarding 3:
What should happen if there already is a file named 'test#2' ?
If something like this is wanted it should cover all eventualities and I need to know whether this is a feature or the file name generation or of the job processing.
To clarify:
During File generation: Existence of file XY would be generated upon job creation. So if you run a job multiple times output would be overwritten.
(atm. 'Generate' option adds a '_new' whenever the file name already exists <- this I would remove)
During job processing: During job processing a Hybrid would check the last 'rename' (or if no 'rename' subjob exists the last processing call to check whether the output exists and adjust the call accordingly - this could be a lot of work.)


Regarding 4:
*.chp (Chapter files) and *.xml (Matroska tag files) should both be created inside the temp folder, not the source folder (unless the source is the temp folder.
+ both should get deleted unless Hybrid is told to keep intermediate files (or the processing is aborted or crashed)
Tested both here and for me the *.chp and the *.xml file both get created inside the temp folder and get deleted during normal processing.
-> If this isn't the case for you, share details and I might be able to fix it.

Cu Selur

(17.11.2017, 06:47)Selur Wrote: [ -> ]Regarding 1.:
This request should go to the x265 developers not me.

I requested it here since this information is mostly already available through Hybrid and the files it produces as long as mkvinfo & mediainfo are up to date. It just takes a while to gather it all manually; except for PSNR & SSIM, as i'm still not sure how to find those values after setting their Metrics on at 'x265 > Misc'.

(17.11.2017, 06:47)Selur Wrote: [ -> ]Regarding 2:
Not going to happen. Intermediate files did at one point contain lots of meta data which causes lots of problems on Windows due to lack of UTF-8 support through a bunch of tools and some Windows combinations. To avoid those problems Hybrid now should use generated names based on the job ID and this is the way it will stay.

I don't think i got the purpose and method of this suggestion properly across...
It would only need to rename the intermediate files (for later reference), after the encoding job is done and full file has been generated. It would only need to use metadata for files supported through mkvinfo & mediainfo. Having a standard source format to read should deal with compatibility problems. If a specified field is not found, it should display nothing and it would be run at the point these files are normally deleted. Readout forwarded to Filename could be limited to supported characters and everything else replaced with something readable.

The real benefit from Suggestions 1 & 2 would be that it would be possible to arrange files by PSNR & SSIM in any file explorer as well as keep track of trends in how .265(, etc.) file's bitrates, sizes and other values correlate with set parameters, essentially turning any file browser into a dynamic comparison chart.

(17.11.2017, 06:47)Selur Wrote: [ -> ]What should happen if there already is a file named 'test#2' ?

What i intend here is that it would always choose the next untaken numerical value and if it reaches #99, it'll go to #100, even if zero padding is set to only 2 digits.

(17.11.2017, 06:47)Selur Wrote: [ -> ]If something like this is wanted it should cover all eventualities and I need to know whether this is a feature or the file name generation or of the job processing.

I would describe it as a quick filesystem check and name correction happening about the same time the job is generating tagging info's, before intermediate files get put into a container.

1. Essentially everything starts with checking whether Generated Filename contains '#{number}', if there is no '#' found, everything is skipped and original file is overwritten.
   
2. When matching files are found for the first time this session, they are listed (internally) and to account for the possibility that user has deleted some of those files (or added a '#00' file), filenames are listed alpha-numerically (number check ends when first non numerical character after # is detected).
   
3. The last filename is used as basis for an availability check: 'generated_hypothetical_filename#'+'detected_number+1' is checked.
   
4. If somehow a new file is generated while this is being checked, +1 is tried, until it's succesful.
   
5. On success the 'new filename' +1 is cached for the next availability check and that check starts at step 3.
   

(17.11.2017, 06:47)Selur Wrote: [ -> ]*.chp (Chapter files) and *.xml (Matroska tag files) should both be created inside the temp folder, not the source folder (unless the source is the temp folder .
+ both should get deleted unless Hybrid is told to keep intermediate files (or the processing is aborted or crashed)

I chose to keep the intermediate (.265 & .aac) files as a workaround, since Hybrid overwrites the .mkv test files i'm generating (as '_new' seems to only kick in occasionally, perhaps due to some condition i'm unaware of; most of my test files only differ by one parameter). I then check their metadata with MediaTab (which internally uses mediainfo).

(17.11.2017, 06:47)Selur Wrote: [ -> ]Tested both here and for me the *.chp and the *.xml file both get created inside the temp folder and get deleted during normal processing.
-> If this isn't the case for you, share details and I might be able to fix it.

This is because i used 'enc\' filename prefix as a workaround in order to get a dynamic subfolder under the source folder. All other temp files get directed to 'enc\' where i want them. I realize i could just set static output and temp paths, but i want to keep my encodes close to my source files. This way i don't get my intermediates from different sources mixed up, nor do i have to jump between tabs as much.

So the point of suggestion 4 is about having full support for dynamic directories for all generated files.

Quote:I requested it here since this information is mostly already available through Hybrid and the files it produces as long as mkvinfo & mediainfo are up to date. It just takes a while to gather it all manually; except for PSNR & SSIM, as i'm still not sure how to find those values after setting their Metrics on at 'x265 > Misc'.

Tell Hybrid to create report files ('Config->Internals->Create report file') and you will see something like this:
Inside the report file. In older versions the normal log showed these, but that was removed since it caused users to always post their normal 'log'.

Problem with generic tags is most applications don't support them.
Most containers like mp4 and mkv, support a few official tags. (For mkv see https://matroska.org/technical/specs/tagging/index.html.)
And libraries like MediaInfo only support these (not sure if MediaInfo really supports all official mkv tags).
-> So in your case it won't help at all if I would add additional mkv tags since MediaInfo won't detect them and report them.

Quote:This is because i used 'enc\' filename prefix as a workaround in order to get a dynamic subfolder under the source folder.

Okay, this feature isn't used as intended and I do not plan to write any workaround for that.

Quote:So the point of suggestion 4 is about having full support for dynamic directories for all generated files.

So a different file directory for each type of generated file. -> Sorry, that's not happening.

Quote:I chose to keep the intermediate (.265 & .aac) files as a workaround, since Hybrid overwrites the .mkv test files i'm generating (as '_new' seems to only kick in occasionally, perhaps due to some condition i'm unaware of; most of my test files only differ by one parameter).

The '_new' will only checked and added when 'Generate' gets triggered.
For example:
You load a file (or multiple files) into Hybrid and 'Generate' is enabled, Hybrid will check for each file whether a name with it's name exists and will add '_new' in case it does. So assuming you now add a job for the file(s) you added, change your encoding settings and create another job:
a. generate won't get triggered
b. no additional '_new' would get added, even if 'Generate' was triggered since the output file doesn't exist.
c. if you do stuff like changing the output folder through file name generation like you did, Hybrid will look inside the wrong folder.

Quote:I then check their metadata with MediaTab (which internally uses mediainfo).

Did some reading up and tested a few things, seems like MediaInfo nowadays supports custom tagging (it even shows non standard tags).
I theoretically could write an option which would add separate tags for any wanted data.
To add PSNR and SSIM results the output of the encoder would need to be parsed and then custom tags would need to be added. (Different code for each multiplexer.)
-> will think about it in case some other users also see a usefulness in this option.

Cu Selur

(17.11.2017, 19:25)Selur Wrote: [ -> ]Tell Hybrid to create report files ('Config->Internals->Create report file') and you will see something like this:

Thanks, this is something i've been looking for, for a while now.

(17.11.2017, 19:25)Selur Wrote: [ -> ]-> So in your case it won't help at all if I would add additional mkv tags since MediaInfo won't detect them and report them. Smile

I haven't actually asked for more tags as such though, but for copying of metadata to output filename. e.g. 'output_filename_bitrate=12345Kbps_size=987654bytes' or 'output_filename_crf=24;qcomp=0.8;pbratio=1.2.mkv'. The exact syntax wouldn't matter either, if ';' or '=' were to cause problems, underscore would do too.

(17.11.2017, 19:25)Selur Wrote: [ -> ]Okay, this feature isn't used as intended and I do not plan to write any workaround for that.

Understandable, but besides the point. I only use it because it's currently closer to what i want than the official implementation for setting directories.

(17.11.2017, 19:25)Selur Wrote: [ -> ]So a different file directory for each type of generated file. -> Sorry, that's not happening.

If i understand you correctly, nope. What i mean is that all intermediate files get put into the same subfolder, that's location is dynamic in relation to either the source or output directory. There is no need to separate each filetype.
Implementations i would find practical would look something like: 'source_folder/encodes/*' with or without the output file, 'output_folder/intermediaries/*' with or without the output file or a new folder named after the output filename (without an extension) such as 'source_folder/output_filename/*' or 'output_folder/output_filename/*'.

(17.11.2017, 19:25)Selur Wrote: [ -> ]The '_new' will only checked and added when 'Generate' gets triggered.
For example:
You load a file (or multiple files) into Hybrid and 'Generate' is enabled, Hybrid will check for each file whether a name with it's name exists and will add '_new' in case it does. So assuming you now add a job for the file(s) you added, change your encoding settings and create another job:
a. generate won't get triggered
b. no additional '_new' would get added, even if 'Generate' was triggered since the output file doesn't exist.

This is consistent with my observations. The way this check works is probably good enough for casual encoding jobs, but when testing and comparing various encoding settings it becomes very inpractical, hence the need to keep the intermediaries, put them in a separate folder, group by type & arrange folder by date. The extra .chp & .xml files in source folder are a rather minor griavance by comparison.

(17.11.2017, 19:25)Selur Wrote: [ -> ]c. if you do stuff like changing the output folder through file name generation like you did, Hybrid will look inside the wrong folder.

Due to points a. and b., knowing this doesn't really make a difference though, hence Suggestion 4.

Btw. if overwriting is a real problem enable 'noOverwriting' through the misc.ini.
see: [INFO] *hidden* Hybrid options,...
This way Hybrid should never overwrite an output file.

Cu Selur

Intermediate files go to the temp folder, unless you tell Hybrid that the temp folder should be the same as the output folder by either selecting:
'Config->Output->Container Settings->Genarl Settings->Keep intermediate files'
or setting:
'Config->Path->Adjust->Temp path to output path'.

Adding another folder which would be used instead of the temp folder seems wrong. That is what the temp folder is for.
-> Adding an option which allows to name a folder that should be used as temp relative to the output folder might solve the problem, but in case the relatively specified temp folder wouldn't exist Hybrid would abort the job creation.

-----------------

About adding data to the output name, it's the same problem as using tags.

• The data has to come from somewhere, meaning the output would have to be parsed and then the tags or the last rename call would need to be adjusted accordingly.
  
• Writing the parser is a pain since:
  • once the encoder changes the output just a little the option would be broken
    
  • each encoder would need a separate parser (since the outputs differ and not all encoders output all values);
    

(17.11.2017, 23:27)Selur Wrote: [ -> ]Btw. if overwriting is a real problem enable 'noOverwriting' through the misc.ini.
see: [INFO] *hidden* Hybrid options,...
This way Hybrid should never overwrite an output file.

Ok, ran some tests and this is somewhat better in that regard, but on the downside it results in very long filenames quite quickly. I'll leave this enabled for now, though.

(18.11.2017, 01:01)Selur Wrote: [ -> ]Adding another folder which would be used instead of the temp folder seems wrong. That is what the temp folder is for.

It would only need to be created when 'Keep intermediate files' is set.

(18.11.2017, 01:01)Selur Wrote: [ -> ]-> Adding an option which allows to name a folder that should be used as temp relative to the output folder might solve the problem, but in case the relatively specified temp folder wouldn't exist Hybrid would abort the job creation.

This would in itself be acceptable for this type of usage scenario, automatically checking for and creating that folder on fly would be even better.
There are however other usage scenarios where setting dynamic paths would also be useful. For example if user were to run Hybrid in portable mode from an USB-stick (as i do) and then move it to another computer... Without dynamic paths you'll need to set these paths manually each time and remember to make sure each directory you set has enough free space available, rather than just checking the free space on the source file's drive.

(18.11.2017, 01:01)Selur Wrote: [ -> ]About adding data to the output name, it's the same problem as using tags.

   The data has to come from somewhere, meaning the output would have to be parsed and then the tags or the last rename call would need to be adjusted accordingly.
   Writing the parser is a pain since:
       once the encoder changes the output just a little the option would be broken
       each encoder would need a separate parser (since the outputs differ and not all encoders output all values);

I guess this will be more of a problem for PSNR & SSIM readouts? In my experience MediaInfo at least seems fairly consistent in it's readouts.
Either way, personally i'd probably implement this using regular expression text searches, like this:
Cache MediaInfo output and search for metadata text readout matching: '[linebreak]File size: ' & copy contents from there on until next [linebreak] is found and remove any spaces. If no matches are found output nothing, if multiple matches are found output longest string by default, if multiple same length strings are found use the first one by default. This way you wouldn't need to separately add support for each possible tag.
It might also be useful to allow user to select one of the other matches and specify a limiter for string length. You could thereby let users specify custom searches using some RegEx like syntax such as:'filename_size=[File size: ]_rate=[{2}Bit rate: (3)]'
With '{2}' being one possible implementation for selecting the second match and '(3)' being one possible implementation of a custom limiter for cutting off excess text where necessary.
Example output in this case would be something like: 'filename_size=12345bytes_rate=123'+.extension
Since this would be a feature for advanced users, when something breaks, normally they'd be able to find out why and correct it for themselves.

PS. Adding PSNR & SSIM to custom tags and/or filename would certainly be a nice feature, but even MediaInfo readouts can be very useful by themselves.
Copying recent changes to 'Encoding settings:' over to filename could be somewhat tricky as well, but probably also the most useful.

Personally i would solve the Encoding settings parsing problem with something like this:

1. On MediaInfo readout, RegEx Replace with nothing: '[linebreak]Encoding settings :' & copy rest of the line.
   
2. Add extra [linebreak] at the very end & Replace every '/' with '[linebreak]' to use as separator for searching subsequent matches.
   
3. RegEx Search for matches from user input using '[space]' as starting and ending character +'[linebreak]', in order to avoid false matches from parameters starting with 'no-' and ending with a multi-digit number.
   
4. When it matches user input RegEx Replace it with nothing.
   
5. RegEx Replace remaining '[linebreak]' characters with ';' and remove the last one.
   
6. RegEx replace every '[space]' with nothing.
   
7. Append the target filename with whatever is left.

Quote:It would only need to be created when 'Keep intermediate files' is set.

Argh. I would prefer to not create folders in Hybrid if at all possible.

-----------------------------------------------------------------------------------------------

Quote:Personally i would solve the Encoding settings parsing problem with something like this:

1. On MediaInfo readout, RegEx Replace with nothing: '[linebreak]Encoding settings :' & copy rest of the line.
   
2. Add extra [linebreak] at the very end & Replace every '/' with '[linebreak]' to use as separator for searching subsequent matches.
   
3. RegEx Search for matches from user input using '[space]' as starting and ending character +'[linebreak]', in order to avoid false matches from parameters starting with 'no-' and ending with a multi-digit number.
   
4. When it matches user input RegEx Replace it with nothing.
   
5. RegEx Replace remaining '[linebreak]' characters with ';' and remove the last one.
   
6. RegEx replace every '[space]' with nothing.
   
7. Append the target filename with whatever is left.
   

This sounds more like something that should be done in a separate tool.
Like a small program (or script) which you feed with a file and a bunch of parameter.

Something like:

Quote:

Code:

MediaInfoRenamer --Inform=<InformCall>  --Separator=<used seperator> --Merger=<append> <File>

Options:

• --Inform: 'Inform'-parameters for MediaInfo. Call "MediaInfo --Info-parameters" to get a full list of supported options.
  
• --Separator: The separator used inside the 'Inform'-parameter.
  
• --Merger: The text that should be used to combine the collected data.
  
• File:: The file which should be analyzed and renamed.
  

For better understanding of what I mean with this.
Here a few examples:

Example 1:
The tool then would:
a. call:
and collect the output.

b. split the output using the separator:

• 640
  
• 352
  
• 722134
  
• 25.000
  

c. join that data using the merger:

d. rename the file from:
to

Example 2:
The tool then would:

a. call:
and collect the output.

b. split the output using the separator:

• 640 pixels
  
• 352 pixels
  
• 722 kb/s
  
• 25.000 FPS
  

c. join that data using the merger:

d. try to rename the file from:
to
and fail since '/' isn't allowed inside a file name.

Note: The following character are reserved and thus will cause problems:

• : (colon)
  
• " (double quote)
  
• / (forward slash)
  
• \ (backslash)
  
• | (vertical bar or pipe)
  
• ? (question mark)
  
• * (asterisk)
  

Example 3:
The tool then would:

a. call:
and collect the output.

b. split the output using the separator:

• Width 640 pixels
  
• Height 352 pixels
  
• Bitrate(kBit pro sec) 722
  
• Framerate 25.000 FPS
  

c. join that data using the merger:

d. rename the file from:
to

Cu Selur

(18.11.2017, 07:54)Selur Wrote: [ -> ]This sounds more like something that should be done in a separate tool.
Like a small program (or script) which you feed with a file and a bunch of parameter.

Something like:

Quote:
MediaInfoRenamer --Inform=<InformCall> --Separator=<used seperator> --Merger=<append> <File>
Options:

--Inform: 'Inform'-parameters for MediaInfo. Call "MediaInfo --Info-parameters" to get a full list of supported options.
--Separator: The separator used inside the 'Inform'-parameter.
--Merger: The text that should be used to combine the collected data.
File:: The file which should be analyzed and renamed.

Hmm... It would also need a way of being pointed towards user's baseline reference for Encoding settings or it won't be able to shorten the output, as well as towards the correct '*_Report.txt' file for PSNR & SSIM parsing if/when implemented here. Other than that this looks good.