Yay Us!

[ndjamena]

I posted this on the VideoHelp Forum, and email Jerome of MediaInfo... Then I almost forgot to post it here:

Hey,

here[1] is a new pre-build. It includes the following two changes you can play around with:

2014-05-20 Moritz Bunkus <moritz@bunkus.org>

* mkvmerge: new feature: When identifying a Matroska file in
verbose identification mode track-specific tags will be output as
well. The format is »tag_<tag name in lower case>:<tag value>»,
e.g. for a tag named »BPS« with the value »224000« the output
would be »tag_bps:224000«. Enhancement for #1021.

* mkvmerge: new feature: mkvmerge will write track-specific tags
with statistics (»BPS« for the average number of bits per second,
»DURATION« for the duration, »NUMBER_OF_BYTES« and
»NUMBER_OF_FRAMES« for the track's size in bytes and its number of
frames/packets). Implements #1021.

Kind regards,
mosu

[1]
https://www.bunkus.org/videotools/mkvto ... -setup.exe

Now I'm wondering how long it will be before MakeMKV joins in the fun!

[mike admin]

Finally. Not so long ago VLC crashed, when any track-specific flag was present at all. This will be added to libmakemkv soon. Any other tags you can think of? Besides the obvious SHA hash?

[ndjamena]

It's preliminary. We've still got to answer the question 'what happens when a program simply copies the tags from one file to another?'

Also, 'If we add another of these new Track Statistic Tags in the future, how will programs know not to copy them into a new encode?'

The answer to both questions may be the same.

[ndjamena]

There's been BPS, FPS and other 'Technical Information' tags in the Matroska tagging system since it's inception. Apparently, no one ever considered the possibility of the things being copied into a new encode until someone tried to do something useful with them.

http://matroska.org/node/22/revisions/22/view

[ndjamena]

My favourite solution at this point is to keep a copy of the segment header's MuxingApp and/or WritingApp elements in each of the tags. If what's in the Tags doesn't match the segment header the tags become invalid. If the tags are written in using a header editor, the header editor would add [AUTH=????] to the segment header element/s and set the Tags to match. ???? would be an arbitrarily generated number, just for added security, but may not me necessary, in fact adding pretty much anything to the segment elements may do. If a legacy header editor comes along and changes the Segment Header, the file automatically reverts to a legacy state and the tags become invalid. New apps would know to remove the AUTH code when displaying the Segment Header elements.

But Moritz know the inner workings of Matroska Files better then I and will most likely come up with a better solution.

There's this point where I could take a hex editor to an MP4 and fill it with all sorts of crap, I don't expect developers to make important decisions based on that fact, you can't cover all eventualities.

-Edit- Damn, if you used a Header Editor to copy tags between two files made with the same program you'd need AUTH numbers in those too. So every file that contains the tags will need a unit security number to validate the tags, hidden somewhere they'd be unlikely to be copied.

[ndjamena]

This is where we're at at the moment:

http://www.bunkus.org/videotools/mkvtoo ... -setup.exe

There are two new tags added _STATISTICS_WRITING_APP and _STATISTICS_WRITING_DATE_UTC for programs to compare with the Segment Header's 'Writing Application' and 'Writing Date' fields to check if the statistics are valid for the current file.

It's at best a hack job, but the objective at the moment is to get these things working in a reliable manner without needing to add new header elements which may play havoc with existing software... again...

Mosu is considering adding another tag that contains the names of all the tags that the _STATISTICS_WRITING_APP and _STATISTICS_WRITING_DATE_UTC tags validate, so the process would be to first check the _STATISTICS_WRITING_APP and _STATISTICS_WRITING_DATE_UTC with the Segment Header, then check the known statistics tags with the list of validated ones and only then use those tags that pass as actual file statistics.

Mosu isn't interested in encrypting the tags or adding a prefix to the statistics tags names. If anyone has a better idea on how to achieve this objective without mangling the file suggestions are welcome.

[ndjamena]

https://flic.kr/p/nKiE4q

Here's a fun fact.

According to the Matroska Website, Tags default to being applied to ALL elements in a file. To change that behaviour you need to add TagTrackUID, TagEditionUID, TagChapterUID or TagAttachmentUID elements to the file. Here's the fun part, if you set one of these UID tags to 0 it automatically selects all file elements of that type, AND THEY ALL DEFAULT TO ZERO, so they're all selected by default, ie if you don't add a particular UID type target, it selects everything of that type. Which means if you set a TagTrackUID to point at a particular track, it will point at that track AND all the Editions AND all the Chapters AND all the attachments EQUALLY. And since these UID tags are unsigned integers, you can only set them to either zero or a positive number, which means there's no way to make a tag that DOESN'T potentially point to at least one attachment, one chapter, one track and one edition. Someone needs to rewrite this or 'track only' tags aren't actually possible.

[ndjamena]

In case anyone is interested, this is the correct use of the MKV Tag Target UIDs:

Code: Select all

<?xml version="1.0"?>
<!-- <!DOCTYPE Tags SYSTEM "matroskatags.dtd"> -->
<Tags>
  <Tag>
    <Targets>
      <TargetTypeValue>70</TargetTypeValue>
      <TargetType>COLLECTION</TargetType>
      <AttachmentUID>15006960677428727970</AttachmentUID>
    </Targets>
    <Simple>
      <Name>TITLE</Name>
      <String>Young Justice (2010)</String>
      <TagLanguage>eng</TagLanguage>
      <DefaultLanguage>1</DefaultLanguage>
    </Simple>
    <Simple>
      <Name>TOTAL_PARTS</Name>
      <String>2</String>
      <TagLanguage>eng</TagLanguage>
      <DefaultLanguage>1</DefaultLanguage>
    </Simple>
    <Simple>
      <Name>CONTENT_TYPE</Name>
      <String>TV Show</String>
      <TagLanguage>eng</TagLanguage>
      <DefaultLanguage>1</DefaultLanguage>
    </Simple>
  </Tag>
  <Tag>
    <Targets>
      <TargetTypeValue>60</TargetTypeValue>
      <TargetType>SEASON</TargetType>
      <AttachmentUID>17160517630404940206</AttachmentUID>
    </Targets>
    <Simple>
      <Name>PART_NUMBER</Name>
      <String>1</String>
      <TagLanguage>eng</TagLanguage>
      <DefaultLanguage>1</DefaultLanguage>
    </Simple>
    <Simple>
      <Name>TOTAL_PARTS</Name>
      <String>26</String>
      <TagLanguage>eng</TagLanguage>
      <DefaultLanguage>1</DefaultLanguage>
    </Simple>
  </Tag>
  <Tag>
    <Targets>
      <TargetTypeValue>50</TargetTypeValue>
      <TargetType>EPISODE</TargetType>
      <AttachmentUID>16402736174017874108</AttachmentUID>
    </Targets>
    <Simple>
      <Name>TITLE</Name>
      <String>Auld Acquaintance</String>
      <TagLanguage>eng</TagLanguage>
      <DefaultLanguage>1</DefaultLanguage>
    </Simple>
    <Simple>
      <Name>PART_NUMBER</Name>
      <String>26</String>
      <TagLanguage>eng</TagLanguage>
      <DefaultLanguage>1</DefaultLanguage>
    </Simple>
    <Simple>
      <Name>DESCRIPTION</Name>
      <String>The Team finally learns the identity of the traitor in their midst, forcing a battle against an enemy more 
dangerous than any they’ve ever faced before!</String>
      <TagLanguage>eng</TagLanguage>
      <DefaultLanguage>1</DefaultLanguage>
    </Simple>
    <Simple>
      <Name>DATE_RELEASED</Name>
      <String>2012-04-21</String>
      <TagLanguage>eng</TagLanguage>
      <DefaultLanguage>1</DefaultLanguage>
    </Simple>
  </Tag>
</Tags>

It assigns Attachments to each of the 3 tag levels (50, 60, 70), basically an image for the episode, an image for the season and an image for the TV Show. You can do the same if you have multiple Level 50 tags. Since I left out any Track, Chapter and Edition UIDs they default to zero which selects all of them. If I run a file containing these tags through FFProbe it spits out this:

Code: Select all

ffprobe version N-63126-gc683e6a Copyright (c) 2007-2014 the FFmpeg developers
  built on May 13 2014 18:06:01 with gcc 4.8.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetyp
e --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-
libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libope
njpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsox
r --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab -
-enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx
--enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxavs --enable-
libxvid --enable-decklink --enable-zlib
  libavutil      52. 83.100 / 52. 83.100
  libavcodec     55. 61.100 / 55. 61.100
  libavformat    55. 37.102 / 55. 37.102
  libavdevice    55. 13.101 / 55. 13.101
  libavfilter     4.  5.100 /  4.  5.100
  libswscale      2.  6.100 /  2.  6.100
  libswresample   0. 18.100 /  0. 18.100
  libpostproc    52.  3.100 / 52.  3.100
Input #0, matroska,webm, from 'F:\Videos\Watch\Young Justice\Young Justice - 01x
01 - Independence Day.mkv':
  Metadata:
    title           : Young Justice - 01x26 - Auld Acquaintance
    encoder         : libebml v1.3.0 + libmatroska v1.4.1
    creation_time   : 2014-06-20 00:47:19
  Duration: 00:21:13.01, start: 0.000000, bitrate: 835 kb/s
    Chapter #0.0: start 0.000000, end 62.160000
    Metadata:
      title           : 00: Intro
    Chapter #0.1: start 62.160000, end 82.560000
    Metadata:
      title           : XX: "Young Justice"
    Chapter #0.2: start 82.560000, end 279.040000
    Metadata:
      title           : 01: Part 1
    Chapter #0.3: start 279.040000, end 597.320000
    Metadata:
      title           : 02: Part 2
    Chapter #0.4: start 597.320000, end 1240.720000
    Metadata:
      title           : 03: Part 3
    Chapter #0.5: start 1240.720000, end 1273.000000
    Metadata:
      title           : XX: Credits
    Stream #0:0(eng): Subtitle: dvd_subtitle, 720x576
    Metadata:
      title           : English - Closed Captions
    Stream #0:1(eng): Video: h264 (High), yuv420p, 1024x576 [SAR 1:1 DAR 16:9],
25 fps, 25 tbr, 1k tbn, 50 tbc (default)
    Metadata:
      title           : AVC/H.264/MPEG4 Part 10 ( fps)
    Stream #0:2(eng): Audio: aac, 48000 Hz, stereo, fltp (default)
    Metadata:
      title           : English AAC(2.0)
    Stream #0:3: Attachment: mjpeg
    Metadata:
      filename        : Cover.jpg
      mimetype        : image/jpeg
      TITLE           : Young Justice (2010)
      TITLE-eng       : Young Justice (2010)
      TOTAL_PARTS     : 2
      TOTAL_PARTS-eng : 2
      CONTENT_TYPE    : TV Show
      CONTENT_TYPE-eng: TV Show
    Stream #0:4: Attachment: mjpeg
    Metadata:
      filename        : Cover.jpg
      mimetype        : image/jpeg
      track           : 1
      PART_NUMBER-eng : 1
      TOTAL_PARTS     : 26
      TOTAL_PARTS-eng : 26
    Stream #0:5: Attachment: mjpeg
    Metadata:
      filename        : Cover.jpg
      mimetype        : image/jpeg
      TITLE           : Auld Acquaintance
      TITLE-eng       : Auld Acquaintance
      track           : 26
      PART_NUMBER-eng : 26
      DESCRIPTION     : The Team finally learns the identity of the traitor in t
heir midst, forcing a battle against an enemy more dangerous than any they???ve
ever faced before!
      DESCRIPTION-eng : The Team finally learns the identity of the traitor in t
heir midst, forcing a battle against an enemy more dangerous than any they???ve
ever faced before!
      DATE_RELEASED   : 2012-04-21
      DATE_RELEASED-eng: 2012-04-21

It has applied the tags only to the attachments, which is completely wrong, because they apply to all the tracks and chapters as well. In fact, they should be listed under the file info, because sadly, under the actual Mastroska Tag specifications it's not actually possible to add tags to an attachment, or a track for that matter and never has been.

[ndjamena]

And if you think simply getting the tags to default to empty will solve the problem, it won't. The tags define titles, and to get it to do anything else without a serious overhaul will simply make it even more complicated then it already is.

[ndjamena]

By the way, I've already brought this up with Moritz Bunkus/Mosu and he agrees with me. In fact I was trying to get him to explain how it actually worked and wasn't really expecting the answer to be the Tags are F@(&3d. I'm annoyed, 'The Tags Define Titles', once you understand that, all the problems we had trying to get the Track Statistics tags in there in the first place finally make sense. The level number is mandatory, yet there's no actual level these tags belong on, the level to put the things on was one of the major sticking points in the discussion and there was no actual logical reasoning behind choosing level 50. I have been kind of annoyed that I've got "MOVIE" tags in my TV Episodes, but let it go because it wasn't important. The Matroska Tags were never designed for any of this in the first place. Some really great programmers have been staring at the specs for god knows how long and no one figured out how they actually worked, in fact far too many programs just assume having a value in a UID target tag targets only that item when it patently is not and never was the case. They didn't read the specs properly. It creates a title that includes that item and all the items of the other types, assuming anything else takes away a lot of the functionality of the system. Any simple fix will cripple the purpose for which the tags were originally intended for, but since no one actually uses it properly or understands it anyway no one will notice until it's too late. Mosu and some other guy created the specs, judging by how much about the tags is currently understood, apparently the other guy did most of the work on them and died shortly afterwards (*joke*).

In summary, The Matroska Tagging System is a bust, I'm annoyed, and I need somewhere to vent.