ssif.smap-to-ssif tool

Discussion of advanced MakeMKV functionality, expert mode, conversion profiles
Post Reply
jemima
Posts: 55
Joined: Fri Oct 06, 2023 1:16 am

ssif.smap-to-ssif tool

Post by jemima » Wed Nov 08, 2023 1:50 am

Hey.

Just in case anyone else would have also a need for this.
Below is a little Python script that generates the *.ssif files from the *.m2ts files using the *.ssif.smap files:

Code: Select all

#!/usr/bin/python3


import sys
import os
import argparse
import re


smap_file_header = "# SSIF MAP FILE v1\n"
default_maximum_read_size = 1024**2




def main():
    #parse command arguments
    parser = argparse.ArgumentParser(allow_abbrev=False)
    parser.add_argument("smap_files", type=str, nargs="+", help="The SMAP-files for which – using the M2TS-files listed therein – the SSIF-files shall be generated in the working directory.", metavar="FILENAME")
    
    args = parser.parse_args()
    
    
    #process SMAP-file operands
    for smap_filename in args.smap_files:
        with open(smap_filename, mode="rt") as smap_file:
            #check SMAP-file header
            line = smap_file.readline( len(smap_file_header) )
            if line != smap_file_header:
                print(f"Error: `{smap_filename}` is not a valid `*.smap`-file.", file=sys.stderr)
                sys.exit(os.EX_DATAERR)
            
            #determine SMAP-file name
            ssif_filename = re.sub(r"\.ssif\.smap$", ".ssif", smap_filename)
            if ssif_filename == smap_filename:
                ssif_filename += ".ssif"
            
            #create SSIF-file
            with open(ssif_filename, mode="xb") as ssif_file:
                #process lines of the SMAP-file
                for line in smap_file:
                    #remove trailing newline (that is: the Line Feed (LF) control character)
                    if line[-1] == "\n":
                        line = line[:-1]
                    
                    
                    #ignore empty and comment lines
                    if re.fullmatch(r"[ \t]*(?:#.*)?", line):
                        continue
                    
                    
                    #parse record
                    m = re.fullmatch(r"(?P<filename>.*) (?P<offset>[0-9a-fA-F]+) (?P<length>[0-9a-fA-F]+)", line)
                    
                    m2ts_filename = m["filename"]
                    offset = int(m["offset"], 16) * 2048
                    length = int(m["length"], 16) * 2048
                    
                    
                    #append range to SSIF-file
                    with open(m2ts_filename, mode="rb") as m2ts_file:
                        m2ts_file.seek(offset)
                        
                        while length > 0:
                            #determine read size
                            if length > default_maximum_read_size:
                                read_size = default_maximum_read_size
                            else:
                                read_size = length
                            
                            
                            #read from the M2TS-file
                            buffer = m2ts_file.read(read_size)
                            
                            buffer_size = len(buffer)
                            length -= buffer_size
                            
                            
                            #check for a premature end of the file
                            if (buffer_size == 0)  and  (length != 0):
                                print(f"Error: Couldn not read {length} bytes from `{m2ts_filename}`.", file=sys.stderr)
                                sys.exit(os.EX_DATAERR)
                            
                            
                            #write buffer
                            ssif_file.write(buffer)
    
    
    sys.exit(os.EX_OK)




if __name__ == "__main__":
    main()




#This program is free software: you can redistribute it and/or modify it under
#the terms of the GNU General Public License as published by the Free Software
#Foundation, either version 3 of the License, or (at your option) any later
#version.
#This program is distributed in the hope that it will be useful, but WITHOUT ANY
#WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
#PARTICULAR PURPOSE.
#See the GNU General Public License for more details.
#You should have received a copy of the GNU General Public License along with
#this program. If not, see <https://www.gnu.org/licenses/>.
There's not much error handling, but it should at least fail noticeable in case of an error (albeit in some cases just by having an exception not caught) and care is taken that no existing files are modified.

HTH,
Jemima

flojo
Posts: 109
Joined: Thu Jun 22, 2023 4:27 am
Location: El Paso

Re: ssif.smap-to-ssif tool

Post by flojo » Wed Nov 08, 2023 1:53 pm

Why would someone need this and what is it?

dcoke22
Posts: 3103
Joined: Wed Jul 22, 2020 11:25 pm

Re: ssif.smap-to-ssif tool

Post by dcoke22 » Wed Nov 08, 2023 4:46 pm

flojo wrote:
Wed Nov 08, 2023 1:53 pm
Why would someone need this and what is it?
viewtopic.php?f=1&t=3455

flojo
Posts: 109
Joined: Thu Jun 22, 2023 4:27 am
Location: El Paso

Re: ssif.smap-to-ssif tool

Post by flojo » Wed Nov 08, 2023 6:59 pm


Application: For the abandoned commercial player "StereoScopic Player".

Maybe for those not familiar with python scoping:

Code: Select all

read_size = length
if read_size > default_maximum_read_size:
    read_size = default_maximum_read_size
BTW I still have no real idea why I would need this. Maybe jemima can add some info on why they wrote it? Synopsis, application, something?

Last edited by flojo on Wed Nov 08, 2023 7:21 pm, edited 2 times in total.

jemima
Posts: 55
Joined: Fri Oct 06, 2023 1:16 am

Re: ssif.smap-to-ssif tool

Post by jemima » Wed Nov 08, 2023 7:16 pm

BTW I still have no real idea why I would need this.
E.g. to recreate the original files as on the BD, in case you want to use that structure with anything other than MakeMKV, which doesn't understand the *.smap files?

flojo
Posts: 109
Joined: Thu Jun 22, 2023 4:27 am
Location: El Paso

Re: ssif.smap-to-ssif tool

Post by flojo » Wed Nov 08, 2023 7:43 pm

jemima wrote:
Wed Nov 08, 2023 7:16 pm
E.g. to recreate the original files as on the BD, in case you want to use that structure with anything other than MakeMKV, which doesn't understand the *.smap files?
O.K., but can you give an example? If I'm someone on the net searching, I'll probably include secondary or tertiary terms. For instance, somebody 12 years ago probably searched for help with something close to "stereoscopic player ssif". Giving as much detail as you can, can help others find your work.

Also, you don't have a usage example :-/

jemima
Posts: 55
Joined: Fri Oct 06, 2023 1:16 am

Re: ssif.smap-to-ssif tool

Post by jemima » Wed Nov 08, 2023 10:24 pm

Uhm I'm not exactly sure what you mean by example and search term?!

The purpose of the program is simply to re-generated the *.ssif files from the *.smap files if someone wants to do that for whichever reason - and if it's just to archive an exact backup of the BD, but not wanting to store the original *.ssif files, but still being able to re-create them.

The program's command line usage should show up with -h or if no arguments are given. It's simply:

Code: Select all

usage: ssif.smap-to-ssif [-h] FILENAME [FILENAME ...]
positional arguments:
  FILENAME    The SMAP-files for which – using the M2TS-files listed therein –
              the SSIF-files shall be generated in the working directory.

options:
  -h, --help  show this help message and exit

h436d
Posts: 9
Joined: Mon Jan 09, 2023 1:05 am

Re: ssif.smap-to-ssif tool

Post by h436d » Tue Jul 23, 2024 4:27 am

jemima wrote:
Wed Nov 08, 2023 10:24 pm
Uhm I'm not exactly sure what you mean by example and search term?!

The purpose of the program is simply to re-generated the *.ssif files from the *.smap files if someone wants to do that for whichever reason - and if it's just to archive an exact backup of the BD, but not wanting to store the original *.ssif files, but still being able to re-create them.

The program's command line usage should show up with -h or if no arguments are given. It's simply:

Code: Select all

usage: ssif.smap-to-ssif [-h] FILENAME [FILENAME ...]
positional arguments:
  FILENAME    The SMAP-files for which – using the M2TS-files listed therein –
              the SSIF-files shall be generated in the working directory.

options:
  -h, --help  show this help message and exit
Hey Jemima,

Fantastic work on this!!
I made the fatal flaw of thinking the folder structure generated by makemkv would allow one to later burn a recovered 3d bluray in the event of the disks being damaged.
Upon further investigation I discovered that UDF 2.5 and up support hard linking and that this is used by SSIF files (as discussed on this post https://forum.videohelp.com/threads/328 ... ost2033751)
Now that I know the ssif.smap files exist and can be used to rebuild the ssif files I'm very pleased in a way.
However the problem remains that unless one plans to burn the original BD50 3d blurays to BDXL 100GB discs, there is no way to build the original UDF 2.5 file structure with the hard links using the makemkv backups.
Are you aware if there is any way to instruct ImgBurn or any other software capable of generating UDF 2.5 iso files, to build the original SSIF files with hard links....
I couldn't find any such tool, which is an aweful shame it seems like even though the backups have all the data required to rebuild the original bluray udf 2.5 file structure, there is no tool to do it...
Perhaps I will need to investigate another backup tool..

UPDATE:
Hopefully this will help someone out.
I think I found a way to rebuild the ssif into a UDF 2.5 compatible iso file.
Basically you need to follow the instructions here.
https://forum.doom9.org/showthread.php? ... ost1652967
"""
7) Use DVDFab Virtual Drive to create mini iso and mount it. This rebuilds the SSIF folder using information in the playlists. The SSIF folder only contains symbolic links to the stream files, which is why you end up with double sized video when you copy this folder to your hard drive.
8) Using any DVD/Blu-ray burning software, such as Imgburn, use the write to image to create the full iso from the mounted mini iso
"""
It requires using DVDFab Virtual Driver with mini-iso that will dynamically rebuild the SSIF files in a way that is UDF 2.5 compatible for iso file generation.
There are more details at the DVDFab site on how mini-iso works.
https://www.dvdfab.cn/miniso.htm

EDIT: Although the miniso files seem to work with dvdfab software, generating an ISO file with imgburn creates iso's that don't seem to work on certain systems. This may not be a fool proof approach but I'm not sure what differs between the generated iso and the original iso that make them incompatible..
Last edited by h436d on Tue Aug 13, 2024 12:43 am, edited 1 time in total.

jemima
Posts: 55
Joined: Fri Oct 06, 2023 1:16 am

Re: ssif.smap-to-ssif tool

Post by jemima » Wed Jul 24, 2024 9:14 pm

h436d wrote:
Tue Jul 23, 2024 4:27 am
I discovered that UDF 2.5 and up support hard linking and that this is used by SSIF files
Hard linking would rather be another name for the same inode (i.e. the whole file). This is more what we in Linux call ref copies or so.
jemima wrote:
Wed Nov 08, 2023 10:24 pm
Are you aware if there is any way to instruct ImgBurn or any other software capable of generating UDF 2.5 iso files, to build the original SSIF files with hard links....
No, I'm afraid I don't. I'd anyway know only about open source tools, but even there I found nothing.

For me it would anyway be much more important to get full support for 3D BluRay to OpenSource, cause only this would give us a truly lasting way to playback 3D BluRays (even if you manage to burn them back to some physical medium - how long will players still be manufactured?).
And the problem there seems to be no 3D support in mkvtoolnix (https://gitlab.com/mbunkus/mkvtoolnix/-/issues/1106) and no MVC support in ffmpeg (https://trac.ffmpeg.org/ticket/3009) ... with no change to that on the horizon. :-(


h436d wrote:
Tue Jul 23, 2024 4:27 am
UPDATE:
Hopefully this will help someone out.
I think I found a way to rebuild the ssif into a UDF 2.5 compatible iso file.
Basically you need to follow the instructions here.
https://forum.doom9.org/showthread.php? ... ost1652967
"""
7) Use DVDFab Virtual Drive to create mini iso and mount it. This rebuilds the SSIF folder using information in the playlists. The SSIF folder only contains symbolic links to the stream files, which is why you end up with double sized video when you copy this folder to your hard drive.
8) Using any DVD/Blu-ray burning software, such as Imgburn, use the write to image to create the full iso from the mounted mini iso
"""
It requires using DVDFab Virtual Driver with mini-iso that will dynamically rebuild the SSIF files in a way that is UDF 2.5 compatible for iso file generation.
There are more details at the DVDFab site on how mini-iso works.
https://www.dvdfab.cn/miniso.htm
Nice finding... but as this depends on closed source, it's only guaranteed to work for now. I mean we've seen how fast something like AnyDVD can disappear... there's unfortunately no guarantee that this cannot happen to DVDFab either.

SamSpeexy
Posts: 3
Joined: Sat Sep 18, 2021 1:26 pm

Re: ssif.smap-to-ssif tool

Post by SamSpeexy » Tue Aug 06, 2024 2:17 am

jemima wrote:
Wed Jul 24, 2024 9:14 pm
h436d wrote:
Tue Jul 23, 2024 4:27 am
UPDATE:
Hopefully this will help someone out.
I think I found a way to rebuild the ssif into a UDF 2.5 compatible iso file.
Basically you need to follow the instructions here.
https://forum.doom9.org/showthread.php? ... ost1652967
"""
7) Use DVDFab Virtual Drive to create mini iso and mount it. This rebuilds the SSIF folder using information in the playlists. The SSIF folder only contains symbolic links to the stream files, which is why you end up with double sized video when you copy this folder to your hard drive.
8) Using any DVD/Blu-ray burning software, such as Imgburn, use the write to image to create the full iso from the mounted mini iso
"""
It requires using DVDFab Virtual Driver with mini-iso that will dynamically rebuild the SSIF files in a way that is UDF 2.5 compatible for iso file generation.
There are more details at the DVDFab site on how mini-iso works.
https://www.dvdfab.cn/miniso.htm
Nice finding... but as this depends on closed source, it's only guaranteed to work for now. I mean we've seen how fast something like AnyDVD can disappear... there's unfortunately no guarantee that this cannot happen to DVDFab either.
I've used the method above and it works with Linux and a Windows virtual machine running with Libvert/QEMU. I copy the blu-ray folder to the virtual machine image after I'm done with MakeMKV and the python script.

Thankfully, there are older 32-bit versions of the DVDFab Virtual Drive program over at Videohelp that work with Windows XP. The program doesn't require an internet connection like AnyDVD does, do there's no need to worry about being left in the cold. Windows XP is the easiest version to set up on a Libvert/QEMU virtual machine, so it's not too difficult to setup and install both IMGBurn and DVDFab Virtual Drive.

h436d
Posts: 9
Joined: Mon Jan 09, 2023 1:05 am

Re: ssif.smap-to-ssif tool

Post by h436d » Tue Aug 13, 2024 12:47 am

SamSpeexy wrote:
Tue Aug 06, 2024 2:17 am
jemima wrote:
Wed Jul 24, 2024 9:14 pm
h436d wrote:
Tue Jul 23, 2024 4:27 am
UPDATE:
Hopefully this will help someone out.
I think I found a way to rebuild the ssif into a UDF 2.5 compatible iso file.
Basically you need to follow the instructions here.
https://forum.doom9.org/showthread.php? ... ost1652967
"""
7) Use DVDFab Virtual Drive to create mini iso and mount it. This rebuilds the SSIF folder using information in the playlists. The SSIF folder only contains symbolic links to the stream files, which is why you end up with double sized video when you copy this folder to your hard drive.
8) Using any DVD/Blu-ray burning software, such as Imgburn, use the write to image to create the full iso from the mounted mini iso
"""
It requires using DVDFab Virtual Driver with mini-iso that will dynamically rebuild the SSIF files in a way that is UDF 2.5 compatible for iso file generation.
There are more details at the DVDFab site on how mini-iso works.
https://www.dvdfab.cn/miniso.htm
Nice finding... but as this depends on closed source, it's only guaranteed to work for now. I mean we've seen how fast something like AnyDVD can disappear... there's unfortunately no guarantee that this cannot happen to DVDFab either.
I've used the method above and it works with Linux and a Windows virtual machine running with Libvert/QEMU. I copy the blu-ray folder to the virtual machine image after I'm done with MakeMKV and the python script.

Thankfully, there are older 32-bit versions of the DVDFab Virtual Drive program over at Videohelp that work with Windows XP. The program doesn't require an internet connection like AnyDVD does, do there's no need to worry about being left in the cold. Windows XP is the easiest version to set up on a Libvert/QEMU virtual machine, so it's not too difficult to setup and install both IMGBurn and DVDFab Virtual Drive.
Hey SamSpeexy,

I am running into issues, where the ISO, I generated with imgburn from the DVDFab Virtual Drive aren't playing back. The miniiso drive plays back fine, but if I make a regular ISO from it, it can't get it to work on certain devices.
Where you able to play the ISO back on certain boxes that support 3d bluray iso like the zidoo, or dvdfabs playerfab player? It looked like the ISO were working at first because they loaded on my standalone bluray player, but then they won't play on certain systems.

EDIT: Also do you mind listing the exact versions of DVDFab VirtualDrive and ImgBurn you are using for generating and converting 3d bluray miniiso? I want to try the exact same versions.
I have a suspicion that DVDFab may be on purpose putting some bad header data in when I try to backup the VirtualDrive to an ISO with ImgBurn, to encourage you to purchase their premium products.
I wonder if using an older version like you, might avoid such a potential countermeasure....

EDIT: It seems like rebooting the system may have fixed the issue. Prior to reboot it was generating ISO from explorefab virtual bluray drive at 10-16x (70MB/s) but the files were unusable. After rebooting it maxes out at 2.2x (12 MB/s) but the ISO file's are usable. I'm not sure what broke, in the pipeline, but I would recommend if anyone is using this technique to convert a collection, one periodically checks if they are actually getting converted properly, loading the bluray for playback is enough to cause things to fail if the ISO is bad based on my experience.

EDIT: Once you manually create miniso files in all the folders, you can automate the remainder of the process using the following commands to mount miniso, generate iso, unmount miniso
```
"C:\\Program Files\\DVDFab\\ExplorerFab\\vdrive.exe" /M:E "z:\\path\\disclabeled_mkvbackupfolder\\dvdfab.miniso"
timeout /t 30
"C:\\Program Files (x86)\\ImgBurn\\imgburn.exe" /mode read /src "E:" /dest "z:\\path\\disclabeled.ISO" /FILESYSTEM "UDF" /UDFREVISION "2.5" /START /CLOSE /WAITFORMEDIA
"C:\\Program Files\\DVDFab\\ExplorerFab\\vdrive.exe" /U:E
timeout /t 30
```

SamSpeexy
Posts: 3
Joined: Sat Sep 18, 2021 1:26 pm

Re: ssif.smap-to-ssif tool

Post by SamSpeexy » Wed Aug 14, 2024 9:42 pm

h436d wrote:
Tue Aug 13, 2024 12:47 am
Hey SamSpeexy,

I am running into issues, where the ISO, I generated with imgburn from the DVDFab Virtual Drive aren't playing back. The miniiso drive plays back fine, but if I make a regular ISO from it, it can't get it to work on certain devices.
Where you able to play the ISO back on certain boxes that support 3d bluray iso like the zidoo, or dvdfabs playerfab player? It looked like the ISO were working at first because they loaded on my standalone bluray player, but then they won't play on certain systems.

EDIT: Also do you mind listing the exact versions of DVDFab VirtualDrive and ImgBurn you are using for generating and converting 3d bluray miniiso? I want to try the exact same versions.
I have a suspicion that DVDFab may be on purpose putting some bad header data in when I try to backup the VirtualDrive to an ISO with ImgBurn, to encourage you to purchase their premium products.
I wonder if using an older version like you, might avoid such a potential countermeasure....

EDIT: It seems like rebooting the system may have fixed the issue. Prior to reboot it was generating ISO from explorefab virtual bluray drive at 10-16x (70MB/s) but the files were unusable. After rebooting it maxes out at 2.2x (12 MB/s) but the ISO file's are usable. I'm not sure what broke, in the pipeline, but I would recommend if anyone is using this technique to convert a collection, one periodically checks if they are actually getting converted properly, loading the bluray for playback is enough to cause things to fail if the ISO is bad based on my experience.
After I create an ISO with IMGBurn, I use IMGBurn's verify mode to ensure that the ISO and the virtual drive are a perfect match. This should help weed out any potential issues.

SamSpeexy
Posts: 3
Joined: Sat Sep 18, 2021 1:26 pm

Re: ssif.smap-to-ssif tool

Post by SamSpeexy » Mon Aug 26, 2024 11:53 pm

I learned about another tool on the Doom9 forums that fits with my current workflow. Xreveal is a Windows ripper/decrypter that works similar to AnyDVD, but uses the keydb database instead. Because it uses keydb, it doesn't require an internet connection to function.

The biggest advantage Xreveal has over the MakeMKV/DVDFab Virtual Drive combo is that it can decrypt discs and rip to ISO if the disc is available in the keydb file. No need to rebuild SSIF files. This helps with 3D discs that use fake playlists or odd forms of seamless branching. Gods of Egypt is the first disc that comes to my mind. It also has 32-bit builds, so I can use it with the Windows XP virtual machine. The downside is that it's not useful for brand new discs, cannot verify M2TS hashes after decryption, and doesn't have a Linux version.

Post Reply