[RFC] DiscImageChef

Discussion of beta and abandonware topics not fit for the other forums goes here.
claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

[RFC] DiscImageChef

Post by claunia »

Greetings programs,

First of all if this post is spam or inappropriate, I'm sorry, I though it may be of interest to beta software collectors.

I've been some years developing DiscImageChef, that you can download from https://github.com/claunia/DiscImageChef

This software has the following features you may find interesting:
  • It is open source
  • It's extensible, so anything not listed here that you need, you can ask or code it yourself
  • It's multiplatform, runs anyplace where you can get .NET/Mono running (tested on Windows, Linux, Mac OS X and FreeBSD)
  • Supports different disc image formats (currently 2MG, BIN+CUE, NRG, TD0, VHD, BIN+TOC, GDI, IMG/ISO/RAW/DD, more coming)
  • Can detect and traverse different partition schemes (Apple Partition Map, Atari, MBR, NeXT disklabel, Amiga RDB, Sun disklabel, GPT)
  • Can detect and show information about several filesystems (Amiga OFS/FFS/FFS2, HFS, HFS+/HFSX, BeFS, ext, ext2, ext3, ext4, UFS/UFS2, HPFS, ISO9660, minix, NTFS, ODS-11, Opera, PC-Engine, ProDOS, SolarOS, System V/Coherent/XENIX, UNIX BFS, Acorn ADFS, btrfs, APFS, Apple Lisa, U.C.S.D Pascal, MFS)
  • Can list and extract files from several filesystems (beta features, supports Apple Lisa, U.C.S.D. Pascal and MFS)
  • Can compare two disc images, even of different formats, for differences
  • Can calculate several checksums of the disc image contents
  • Can verify a disc image integrity (if the format or the contents support it)
  • Can output sectors in hexadecimal
  • Can create an XML sidecar with information describing the disc image and its contents
  • Can get information from an IDE/EIDE/ATAPI/SATA, SCSI/SAS, USB, FireWire device (tape or disk, only supported on Linux and Windows)
  • Can get information from the media inserted on a device
  • Can dump a disk (anything attached in the previously named buses. Tapes not yet supported)
I'm posting it here for your joy, in the hope you spam me with feature requests and bug reports.

Thanks for reading

Overdoze
User avatar
FTP Access
Posts: 1695
Joined: Mon Feb 24, 2014 10:28 am
Location: Slovenia

Re: [RFC] DiscImageChef

Post by Overdoze »

This actually sounds pretty good, I'll definitely try it out. Thanks for posting this. :)
All roads lead to Neptune™

KRNL386 - my site about retro computing | My site about Windows 1.0 | My blog | 86Box Manager | LeakDB - list of PC OS warez leaks

hounsell

Re: [RFC] DiscImageChef

Post by hounsell »

This is easily one of the most useful projects posted to BA in years. I can definitely see this coming in handy for some of my own projects.

Ahmed Jebara
Donator
Posts: 1559
Joined: Mon Jul 19, 2010 8:34 pm

Re: [RFC] DiscImageChef

Post by Ahmed Jebara »

Now that's genuinely a nice job. Well done!
%1

Overdoze
User avatar
FTP Access
Posts: 1695
Joined: Mon Feb 24, 2014 10:28 am
Location: Slovenia

Re: [RFC] DiscImageChef

Post by Overdoze »

Also, I like that little Tron reference. :)
All roads lead to Neptune™

KRNL386 - my site about retro computing | My site about Windows 1.0 | My blog | 86Box Manager | LeakDB - list of PC OS warez leaks

marcoguy
User avatar
Donator
Posts: 178
Joined: Sat Jun 16, 2012 1:28 am
Location: PATH=C:\DOS

Re: [RFC] DiscImageChef

Post by marcoguy »

Wow, when I initially saw this thread I was expecting another kid wanting feedback on the "web browser" he made with VB/WinForms or something, but I couldn't have been more wrong :)

Really great work. As Hounsell said, this is the most useful project I've seen here in a long time.

Just wondering, do you plan on implementing support for the Alcohol 120℅ MDS/MDF format? It's what I use to dump CDs and DVDs in my collection, so I don't know about anyone else, but I'd certainly find it useful.
Preservation
How could you know where you want to go today if you don't even know where you've been?

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

marcoguy, I plan to support all disc image formats I can.

Overdoze
User avatar
FTP Access
Posts: 1695
Joined: Mon Feb 24, 2014 10:28 am
Location: Slovenia

Re: [RFC] DiscImageChef

Post by Overdoze »

Could support for CopyQM images (by Sydex, makers of TeleDisk) be added?
All roads lead to Neptune™

KRNL386 - my site about retro computing | My site about Windows 1.0 | My blog | 86Box Manager | LeakDB - list of PC OS warez leaks

Darkstar
User avatar
Donator
Posts: 919
Joined: Fri May 14, 2010 1:29 pm
Location: Southern Germany

Re: [RFC] DiscImageChef

Post by Darkstar »

Awesome! I especially like your priorities (FILES-11 and Sun disklabels for example ;-) )

Started and watching the Repo now (have some ideas already :-) )
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

Overdoze wrote:Could support for CopyQM images (by Sydex, makers of TeleDisk) be added?
Added on https://github.com/claunia/DiscImageChe ... /v3.2.99.1

But I only had a few images to test, XENIX 2.3.2d and some synthetizer disks so if it crashes, report :p

marcoguy, Alcohol 120% will come soon, do not worry.

Darkstar feel free to PM your ideas or create an issue in Github.

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

marcoguy wrote:Wow, when I initially saw this thread I was expecting another kid wanting feedback on the "web browser" he made with VB/WinForms or something, but I couldn't have been more wrong :)

Really great work. As Hounsell said, this is the most useful project I've seen here in a long time.

Just wondering, do you plan on implementing support for the Alcohol 120℅ MDS/MDF format? It's what I use to dump CDs and DVDs in my collection, so I don't know about anyone else, but I'd certainly find it useful.
Added to https://github.com/claunia/DiscImageChe ... /v3.2.99.2
Enjoy :P !

TheJetixMasta
User avatar
Posts: 2
Joined: Sat Apr 29, 2017 12:56 am

Re: [RFC] DiscImageChef

Post by TheJetixMasta »

This is actually very useful. Thanks for posting! I'll definitely check it out! :D
I love using beta software and normally install a few betas on Virtualbox.

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

I'm happy to see people find my tool so useful as to make this a sticky topic.

You really have my gratitude!

I have been preparing a HUGE update with lots of new features, and as soon as I have it prepared (as I'm doing this alone in my free time) I will post the details here.

Nonetheless don't forget, please, any feature you want, any bug you discover, any disk/disc/tape image format you want supported, create an issue in github and I'll check it as soon as I can.

tristanleboss
FTP Access
Posts: 63
Joined: Wed Jan 11, 2017 12:37 pm

Re: [RFC] DiscImageChef

Post by tristanleboss »

I just wanted to add my 2 cents: Of course your tool (toolbox?) is a great and useful one ;)

Congratulations for your work! I hope you will find the time to work on it... I know it's not easy to maintain such a project aside from a real job and a real life ;)

mrpijey
User avatar
Administrator
Posts: 8180
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: [RFC] DiscImageChef

Post by mrpijey »

Would be great if someone could do a nice technical comparison between Alcohol120% and DiscImageChef and list the pros and cons of both, how easy the formats are used on Win and OSX etc... it's a very interesting project indeed.
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

tristanleboss
FTP Access
Posts: 63
Joined: Wed Jan 11, 2017 12:37 pm

Re: [RFC] DiscImageChef

Post by tristanleboss »

DIC is way more complete than Alcohol120%. It can work with any type of devices and have many more features which are not part of Alcohol120%.

The only feature that can be compared to one from Alcohol120% would be the `dump-media` one.

Battler
User avatar
Donator
Posts: 2034
Joined: Sat Aug 19, 2006 8:13 am
Location: Slovenia, Central Europe.
Contact:

Re: [RFC] DiscImageChef

Post by Battler »

Does DiscImageChef support data position measurement (preferably at high precision)? It's needed for some games like the Italian second edition of Who Wants To Be A Millionaire for example.
Join [url=irc://irc.ringoflightning.net/softhistory/]#softhistory @ RoL IRC[/url], a nice community for true enthusiasts!
Anime channel: [url=irc://irc.ringoflightning.net/aniboshi/]#doki-doki @ RoL IRC[/url], Mibbit, KiwiIRC.
The 86Box help channel is #softhistory now!

Check out our SoftHistory Forum for quality discussion about older software.

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

Battler wrote:Does DiscImageChef support data position measurement (preferably at high precision)? It's needed for some games like the Italian second edition of Who Wants To Be A Millionaire for example.
No, sorry, it's a feature I have not explored and I lack the media to test it :/

Edit: If you can express the algorithm to do it in C, or know an opensource implementation of the algorithm I can add it.

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

tristanleboss wrote:DIC is way more complete than Alcohol120%. It can work with any type of devices and have many more features which are not part of Alcohol120%.

The only feature that can be compared to one from Alcohol120% would be the `dump-media` one.
In a nutshell, not an extensive comparison, just from memory:

What can Alcohol120% do and DiscImageChef don't:
  • Data Position Measurement
  • Dump multi-track CDs in a format mountable by emulators (I will solve this as soon as I can)
What can DiscImageChef do and Alcohol120% don't:
  • Dump Xbox Game Discs
  • Dump floppies (with USB drives only)
  • Dump SCSI tapes
  • Dump PCMCIA/CompactFlash saving card identification registers
  • Dump SCSI/ATA/SATA/USB/FireWire devices
  • Resume where you left the dump
  • Retry errors in another moment (after polishing?), or even another (less-picky?) drive
  • Read RAW DVD frames (in Matshita drives, requires disc swap for DVD-ROM, decoding is to be implemented)
  • Read LONG sectors from most SCSI devices and superfloppies (LS120/ZIP/JAZ/etc)
  • Works on Linux
  • Does not need GUI
  • Can be scriptable for unattended dump
  • Analyzes dumps in lots and lots of formats, identifies filesystems, checksums dump files and their contents, generates XML metadata
  • It's opensource, you can audit, enhance, or add any feature you want
So yeah there are some important things DiscImageChef doesn't, just tell me and I'll implement them as soon as I can.
Also being opensource, if you have the technical expertise to implement it yourself, you're absolutely welcome to do so.

mrpijey
User avatar
Administrator
Posts: 8180
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: [RFC] DiscImageChef

Post by mrpijey »

claunia wrote: In a nutshell, not an extensive comparison, just from memory:

What can Alcohol120% do and DiscImageChef don't:
  • Data Position Measurement
  • Dump multi-track CDs in a format mountable by emulators (I will solve this as soon as I can)
What can DiscImageChef do and Alcohol120% don't:
  • Dump Xbox Game Discs
  • Dump floppies (with USB drives only)
But not DMF floppies I guess....?
claunia wrote:[*]Dump SCSI tapes
[*]Dump PCMCIA/CompactFlash saving card identification registers
[*]Dump SCSI/ATA/SATA/USB/FireWire devices
Which A120 can do too. Or what do you mean with "devices"? What devices?
claunia wrote:[*]Resume where you left the dump
[*]Retry errors in another moment (after polishing?), or even another (less-picky?) drive
[*]Read RAW DVD frames (in Matshita drives, requires disc swap for DVD-ROM, decoding is to be implemented)
[*]Read LONG sectors from most SCSI devices and superfloppies (LS120/ZIP/JAZ/etc)
[*]Works on Linux
[*]Does not need GUI
[*]Can be scriptable for unattended dump
[*]Analyzes dumps in lots and lots of formats, identifies filesystems, checksums dump files and their contents, generates XML metadata
[*]It's opensource, you can audit, enhance, or add any feature you want[/list]

So yeah there are some important things DiscImageChef doesn't, just tell me and I'll implement them as soon as I can.
Also being opensource, if you have the technical expertise to implement it yourself, you're absolutely welcome to do so.
Sounds like it can do a lot of stuff. What it should be able to do is also dump to some standard (and semi-standard) formats... ISO, BIN/CUE, DMG, MDF/MDX.... does it have a logging option to save a log with the dump? Any way to extract the dumped images? (For example, if I made an image of a ZIP floppy and want to access the files...).

Keep up the good work! If it keeps up it may become a format we can accept if it has all the critical low level stuff in it... Any OSX client?
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

mrpijey wrote:
claunia wrote: In a nutshell, not an extensive comparison, just from memory:

What can Alcohol120% do and DiscImageChef don't:
  • Data Position Measurement
  • Dump multi-track CDs in a format mountable by emulators (I will solve this as soon as I can)
What can DiscImageChef do and Alcohol120% don't:
  • Dump Xbox Game Discs
  • Dump floppies (with USB drives only)
But not DMF floppies I guess....?
All USB floppies afaik support standard MF2DD and MF2HD disks in DOS format.
MOST USB floppies support NEC 2HD disks.
SOME USB floppies support DMF disks.

Unfortunately that depends on the drive itself and there is nothing I can do software side.

LS120/LS240 drives support MF2DD, MF2HD and NEC 2HD disks. I have not tested with DMF disks.
mrpijey wrote:
claunia wrote:[*]Dump SCSI tapes
[*]Dump PCMCIA/CompactFlash saving card identification registers
[*]Dump SCSI/ATA/SATA/USB/FireWire devices
Which A120 can do too. Or what do you mean with "devices"? What devices?
Hard disks, SSDs, streaming tapes (QIC, LTO, DAT, DDS, AIT, etc), flash drives, PD650, ZIP100, ZIP250, ZIP750, SCSI or ATAPI floppies (MF2DD, MF2HD and MF2ED drives, floptical, LS120/LS240 floppies, HiFD), USB Mass Storage Devices (flash drives, ATAPI-USB/ATA-USB bridges), PCMCIA ATA cards, CompactFlash (to get card descriptors connect with a PCMCIA-CompactFlash adapter, USB readers block access to that information). In theory, anything that attaches to ATA, SATA, SCSI, FireWire or USB but DV tapes.
mrpijey wrote:
claunia wrote:[*]Resume where you left the dump
[*]Retry errors in another moment (after polishing?), or even another (less-picky?) drive
[*]Read RAW DVD frames (in Matshita drives, requires disc swap for DVD-ROM, decoding is to be implemented)
[*]Read LONG sectors from most SCSI devices and superfloppies (LS120/ZIP/JAZ/etc)
[*]Works on Linux
[*]Does not need GUI
[*]Can be scriptable for unattended dump
[*]Analyzes dumps in lots and lots of formats, identifies filesystems, checksums dump files and their contents, generates XML metadata
[*]It's opensource, you can audit, enhance, or add any feature you want[/list]

So yeah there are some important things DiscImageChef doesn't, just tell me and I'll implement them as soon as I can.
Also being opensource, if you have the technical expertise to implement it yourself, you're absolutely welcome to do so.
Sounds like it can do a lot of stuff. What it should be able to do is also dump to some standard (and semi-standard) formats... ISO, BIN/CUE, DMG, MDF/MDX.... does it have a logging option to save a log with the dump? Any way to extract the dumped images? (For example, if I made an image of a ZIP floppy and want to access the files...).
For CDs it dumps 2448 bytes/sector, all sequential in a file. I need to add CUE creation to know where a track starts.
For anything else, it dumps sectors as is, what you call ISO/IMG basically.

If you use --raw it depends on hardware, e.g.:
DVDs on Matshita drives: It outputs raw 37856 blocks sequentially.
Floppies on LS drives: It outputs 512 data bytes + 4 CRC bytes sequentially.
ZIP drives: It outputs 512 data bytes + xx ECC bytes sequentially.

Non-user-data will be written in separate files

But in a nutshell, from a DVD drive:
DiscImageChef dump-media -i /dev/sr0 -w mydisk
Will create mydisk.iso, mydisk.pfi.bin (for the DVD PFI), mydisk.dmi.bin (for the DVD DMI).
mrpijey wrote:Keep up the good work! If it keeps up it may become a format we can accept if it has all the critical low level stuff in it...
For non-copy protected media, I think it already does :p
mrpijey wrote:Any OSX client?
The problem with macOS itself is that it only allows you to send custom command to optical disc recorders. So it's absolutely useless for any other thing.
And even then, it blocks some commands, and it's a pain in the ass to develop.
If you want to dump media on a Mac, as you'll be using a USB adaptor or the internal recorder, it's best you launch up a virtualbox instance with Linux (I have DIC available in a bootable flash drive that can be used to boot VBox or directly on hardware) and pass-thru the device :/

All the other functions that don't access directly to a device (that's, all but dump-media, device-info and media-info) work the same on macOS, Windows, Linux and FreeBSD.

mrpijey
User avatar
Administrator
Posts: 8180
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: [RFC] DiscImageChef

Post by mrpijey »

Well I guess it always boils down to OSX's limited usability which means I have to stop counting on finding a format that works with OSX, and everyone has to rely on Linux and Windows systems to handle the images...

I think you should have some kind of image descriptor for your stuff, something that clearly explains how the image is dumped, sector size, tracks etc etc. The more info your tool can save about the disc and its format the better for preservation purposes, but also for the sake of being able to reproduce the disc. DPM support is a must of course, dumping floppies with a real floppy drive should also be supported, logging proper multilayer breaks in dual layer DVDs and multilayer BDROMs etc should also be recorded somehow. How is it with other discs such as certain console discs? You mentioned it does Xbox, what about newer Xbox stuff? Sega Dreamcast? Nintendo Gamecube and Wii? Etc etc...
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

mrpijey wrote:Well I guess it always boils down to OSX's limited usability which means I have to stop counting on finding a format that works with OSX, and everyone has to rely on Linux and Windows systems to handle the images...

I think you should have some kind of image descriptor for your stuff, something that clearly explains how the image is dumped, sector size, tracks etc etc. The more info your tool can save about the disc and its format the better for preservation purposes, but also for the sake of being able to reproduce the disc. DPM support is a must of course, dumping floppies with a real floppy drive should also be supported, logging proper multilayer breaks in dual layer DVDs and multilayer BDROMs etc should also be recorded somehow. How is it with other discs such as certain console discs? You mentioned it does Xbox, what about newer Xbox stuff? Sega Dreamcast? Nintendo Gamecube and Wii? Etc etc...
DPM I must learn how to do it. Xbox 360, Dreamcast, Gamecube and Wii are W.I.P.. PS3, XboxOne and WiiU I don't have the drives. All the information you say that should be stored, is, in the metadata. It's just I don't consider the metadata to be a "format".

This is the dump from the Lemote Fuulong system disk containing Debian/mips:

Code: Select all

claunia@zeus /mnt/Dumps $ /bin/ls -l lemote.*
-rw-r--r-- 1 claunia users       7740 feb 13  2016 lemote.cicm.xml
-rw-r--r-- 1 claunia users          4 feb 13  2016 lemote.cmi.bin
-rw-r--r-- 1 claunia users       2048 feb 13  2016 lemote.dmi.bin
-rw-r--r-- 1 claunia users     126626 feb 13  2016 lemote.ibg
-rw-r--r-- 1 claunia users 4068376576 feb 13  2016 lemote.iso
-rw-r--r-- 1 claunia users     496803 feb 13  2016 lemote.mhddlog.bin
-rw-r--r-- 1 claunia users       2048 feb 13  2016 lemote.pfi.bin
Contents of lemote.cicm.xml: https://pastebin.com/rRUif9yu

Command line used:

Code: Select all

mono DiscImageChef.exe dump-media -i /dev/sr0 -w lemote
lemote.dmi.bin contains the DVD DMI.
lemote.cmi.bin contains the DVD Lead-In CMI.
lemote.pfi.bin contains the DVD PFI.
lemote.mhddlog.bin contains a read log in MHDD format (viewers available online).
lemote.ibg contains a read/speed log in Imgburn format (viewer included with imgburn).
lemote.iso is, well, an ISO :p

tristanleboss
FTP Access
Posts: 63
Joined: Wed Jan 11, 2017 12:37 pm

Re: [RFC] DiscImageChef

Post by tristanleboss »

The thing is: how to put all these data back to a disk? Preserving these data is important but being able to recreate the media is what make a tool usable for archivism (like the Kryoflux which can rewrite its raw files).

With Alcohol12%, it's easy: you have 2 file and the software can reread them.

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

tristanleboss wrote:The thing is: how to put all these data back to a disk? Preserving these data is important but being able to recreate the media is what make a tool usable for archivism (like the Kryoflux which can rewrite its raw files).

With Alcohol12%, it's easy: you have 2 file and the software can reread them.
Dumping in formats that burning software will understand is something I will add in the next-to-next version (5.0).
Converting formats (e.g. CopyQM to Teledisk, Nero to Alcohol) is something I will add in that same version.
Making DiscImageChef write media is something I don't pretend to add, but I will accept any patch doing it.

Post Reply