[RFC] DiscImageChef

Discussion of beta and abandonware topics not fit for the other forums goes here.
mrpijey
User avatar
Administrator
Posts: 8183
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: [RFC] DiscImageChef

Post by mrpijey »

Well, one critical part about every preservation tool is being able to recreate the image onto the format it originated from, so I hope you'll add that functionality in the future. Or at least being able to convert to a format so we can use other tools.
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

tristanleboss
FTP Access
Posts: 63
Joined: Wed Jan 11, 2017 12:37 pm

Re: [RFC] DiscImageChef

Post by tristanleboss »

I agree with mrpijey.

Having your own dumping format is cool (and you should keep it... it's a bit like the Kryoflux raw one) but if your tool can -for example (and like the dtc Kryoflux tool)- create the exact same MDF/MDS file from a DVD (or from a set of raw files) than Alcohol 120%, it will be really interesting. Because it will mean your dump will be usable easily. Version 5.0 will be the interesting one.

Dumping floppy from a real 3.5" drives or 5.25" would be cool to. This way it will be able to replace many smaller tools.

Anyway, I can understand - due tot he variety of functions offered by your tool - that it targets different types of users with different needs.

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

tristanleboss wrote:I agree with mrpijey.

Having your own dumping format is cool (and you should keep it... it's a bit like the Kryoflux raw one) but if your tool can -for example (and like the dtc Kryoflux tool)- create the exact same MDF/MDS file from a DVD (or from a set of raw files) than Alcohol 120%, it will be really interesting. Because it will mean your dump will be usable easily. Version 5.0 will be the interesting one.

Dumping floppy from a real 3.5" drives or 5.25" would be cool to. This way it will be able to replace many smaller tools.

Anyway, I can understand - due tot he variety of functions offered by your tool - that it targets different types of users with different needs.
I won't use my own dumping format, even if I designed one extensible format, because that would mean only my tool supports it.
For CDs I will use BIN/CUE and for DVDs/BDs MDF/MDS.
For other devices there is no need to do anything, it already creates IMG.

Dumping floppies from real drives can be done on three ways:
1.- Using an USB floppy drive. This is already supported, but because of the drives themselves, it's limited to non-copy-protected disks in a handful of formats.
2.- Using a PC floppy controller. This is doable, but don't expect it anytime soon.
3.- Using a kryoflux/discferret/supercardpro. This is huge problem. KryoFlux is secretive, everything but what conservation means, yet it's the most common hardware, so I would need to reverse engineer their USB protocol (and this, takes a LOT of time). DiscFerret and SuperCardPro should be easier to do, their not as secretive, but I don't have the hardware to test it, and without it, I just wouldn't be able to support them.

In a perfect world, where everything goes as planned, you'll see these come alive:
BIN/CUE and MDF/MDS in DiscImageChef 4.0 (planned in a few months)
Image format conversion in DiscImageChef 5.0 (Xmas '17)
Supporting kryoflux images in DiscImageChef 5.5 (2018)
PC floppy controller in DiscImageChef 5.5 (2018)
Creating images from a kryoflux device in DiscImageChef 6.0 (2019 or 2020)
Creating images from a discferret or supercardpro device in DiscImageChef 5.5 if someone donates me the controllers, if not, probably never for discferret (as it's no longer manufactured) and who knows when for supercardpro.

Darkstar
User avatar
Donator
Posts: 920
Joined: Fri May 14, 2010 1:29 pm
Location: Southern Germany

Re: [RFC] DiscImageChef

Post by Darkstar »

claunia wrote:3.- Using a kryoflux/discferret/supercardpro. This is huge problem. KryoFlux is secretive, everything but what conservation means, yet it's the most common hardware, so I would need to reverse engineer their USB protocol (and this, takes a LOT of time). DiscFerret and SuperCardPro should be easier to do, their not as secretive, but I don't have the hardware to test it, and without it, I just wouldn't be able to support them.
The KryoFlux USB protocol is probably a very simple one. I'd suggest running Wireshark on your USB port while dumping a disk or two, and seeing what you get. My guess it's that it's rather easy to replicate. But I might be wrong.

If you want, I can provide you with some USB traces and the corresponding command line / raw dumps for a handful of floppies, but at the moment I don't have enough free time to tinker with reverse engineering the protocol myself, sadly
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

Darkstar wrote:
claunia wrote:3.- Using a kryoflux/discferret/supercardpro. This is huge problem. KryoFlux is secretive, everything but what conservation means, yet it's the most common hardware, so I would need to reverse engineer their USB protocol (and this, takes a LOT of time). DiscFerret and SuperCardPro should be easier to do, their not as secretive, but I don't have the hardware to test it, and without it, I just wouldn't be able to support them.
The KryoFlux USB protocol is probably a very simple one. I'd suggest running Wireshark on your USB port while dumping a disk or two, and seeing what you get. My guess it's that it's rather easy to replicate. But I might be wrong.

If you want, I can provide you with some USB traces and the corresponding command line / raw dumps for a handful of floppies, but at the moment I don't have enough free time to tinker with reverse engineering the protocol myself, sadly
Darkstar I already stored one USB trace. I don't think there's need for more than one, it's streaming, dtc just asks kryoflux for data and everything is streamed over usb. Nonetheless if you create more traces, better yet.
The problem is I don't have the time myself either :/

Maybe there's people interested on paying our time on a bounty? I do not move in those forums but from what I've heard, lots of people want to be able to use the kryoflux with a raspberry pi, something their creators have repeatedly said they won't support, but DiscImageChef will have no problem doing.

mrpijey
User avatar
Administrator
Posts: 8183
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: [RFC] DiscImageChef

Post by mrpijey »

I think we should leave the actual dumping to the KF team as there's more to it than just streaming USB data... but conversion and reading support could be implemented using their libraries.
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

mrpijey wrote:I think we should leave the actual dumping to the KF team as there's more to it than just streaming USB data... but conversion and reading support could be implemented using their libraries.
There's a problem, their libraries.

See, KryoFlux just streams the pulses from the drive and the internal clock thru USB. They even call their output STREAM, that they don't document.
Their software, aka dtc, stores it, and for some formats, creates a RAW image, that, btw loses all tags for Lisa and MacOS double density disks (MFS filesystem uses the tags for filesystem recovery, LisaOS requires it for reading the disk). A lot of disks (XDF, copy protected) just don't work.
Then they want you to send them the STREAM to create, yet another closed source undocumented format called IPF using a software that they sell under NDA (so far for software preservation). Originally they provided only binary blobs, later they provided source code under a license that is not opensource, neither OSI neither GNU compliant. You can read it here: https://github.com/lunixbochs/fs-uae-gl ... ge/LICENSE
This by itself prevents using their libraries (even looking at the code in that folder may damage DiscImageChef development).

That's why people like Keirf (https://github.com/keirf/Disk-Utilities/) and Jean Louis (http://info-coach.fr/atari/software/preservation.php) created their own utilities to convert the STREAM files to appropriate files, being RAW or files that support copy protection for specific emulators.

Adding support for KryoFlux STREAM files, and the device itself, is a must, because it's so extended, people is creating the need to depend on a group that's closed, secretive, and everything but preserving.
Also it's by far the more closed, least powerful, and least recommended of the equivalent devices.

DiscFerret is fully opensource, and powerful enough to dump even hard disks, but unfortunately abandoned by its creator it cannot be bought anymore.
SuperCardPro is not as opensource as DiscFerret, but their format don't need to be hacked (they provide full specifications) and their protocol neither (also, documentation provided).

More open -> more propietary
DiscFerret -> SuperCardPro -> KryoFlux

mrpijey
User avatar
Administrator
Posts: 8183
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: [RFC] DiscImageChef

Post by mrpijey »

And yet it's the first hit I found when I looked for it...

https://www.kryoflux.com/download/kryof ... rev1.1.pdf

Oh look, another one for IPF

http://info-coach.fr/atari/documents/_m ... tation.pdf

Sure the second one is not directly from source devs, but looks pretty well documented to me. Quite a few tools and emulators support it as well, so it must be possible to implement support for it one way or another...

I do however agree on SPS's closed "preservation" tactics which are anything but preservation, which is why I stopped donating stuff to them. No point of sharing stuff if the point of failure can be traced down to a single place...

KryoFlux seems to be the more used and preferred tool as well, a lot due to the better hardware and more open and active community around it. I never had issued with it, written back a lot of floppies from the STREAM format without any issues whatsoever, and the software is just better... (yes I own an SCP as well, and its software is dodgy at best).

What I am saying is that your tool doesn't need to support the hardware devices directly, only the formats so it can convert from them (if you intend your tool to also be able to read from one format and write to another, if not then ignore everything I've said).

Edit: But the main question comes down to: What are your intentions with your software? If you want to have preservation quality features in it then it need to
  1. support other formats such as STREAM, IPF, SCP and other low level formats, at least be able to read from them and convert.
  2. support full read and write of your own format if you choose to use one of your own
  3. be fully documented, at least to the extent that others can use your own format.
  4. support the basic storage hardware, such as floppy drives and (S/P)ATA optical drives.
  5. read the media at its lowest possible level to preserve as much info as possible.
If you don't intend it to be a tool for preservation but just one to read various storage media then fine, but then it's not suitable for preservation either and should not be used as such. There are already so many tools out there that handles the various formats, but not one that does all of it well. Considered to make it modular? Allow a plugin system where others can develop their own plugins to support all kinds of stuff and then lift some of the pressure off you.

But the important thing is that you can read and write your own format. A tool that can't use its own format to recreate the original is pretty much useless, since what use do you have of a bunch of image files if they are unusable? It's like storing a picture in a format no other applications can handle... And if you decide to only support the high level stuff such as ISO, BIN/CUE etc then that's fine, but for BetaArchives preservation purposes it won't be enough.

Don't get me wrong, I am very interested in your software and it looks like a very nice candidate for an all-in-one media dumper, but it has to be able to preserve as much info as possible of a media, and do it in a way that can either be recreated (using your software) or dumped into a format other low level tools can handle, such as KryoFlux RAW/STREAM, MDF/MDS/MDX, CloneCD etc, so it can be mountable and usable outside your tool.

Keep up the good work!
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

Darkstar
User avatar
Donator
Posts: 920
Joined: Fri May 14, 2010 1:29 pm
Location: Southern Germany

Re: [RFC] DiscImageChef

Post by Darkstar »

claunia wrote:Adding support for KryoFlux STREAM files, and the device itself, is a must, because it's so extended, people is creating the need to depend on a group that's closed, secretive, and everything but preserving.
Also it's by far the more closed, least powerful, and least recommended of the equivalent devices.

DiscFerret is fully opensource, and powerful enough to dump even hard disks, but unfortunately abandoned by its creator it cannot be bought anymore.
SuperCardPro is not as opensource as DiscFerret, but their format don't need to be hacked (they provide full specifications) and their protocol neither (also, documentation provided).
I agree with you that adding support for STREAM files is a must. But I don't think the same of their dumping hardware. STREAM files are a perfectly fine medium for lossless transportation of floppy dumps, no need to support the physical KF device. It would be nice to have, yes. But definitely not required.

It's a shame that KF is closed-source, I agree, but let's face it, it's simply the bestsolution out there for dumping floppy disk with their copy protection intact. The guys behind KF are long-time experts in the field of floppy disk technology, the single guy behind SCP is an individual who is not the easiest person around to work with (from what I heard... I don't know Jim personally so take my opinion with a grain of salt).

There is probably a reason almost every institution (libraries, museums, etc.) are using a KF instead of the SCP... ;-)
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

Darkstar wrote:
claunia wrote:Adding support for KryoFlux STREAM files, and the device itself, is a must, because it's so extended, people is creating the need to depend on a group that's closed, secretive, and everything but preserving.
Also it's by far the more closed, least powerful, and least recommended of the equivalent devices.

DiscFerret is fully opensource, and powerful enough to dump even hard disks, but unfortunately abandoned by its creator it cannot be bought anymore.
SuperCardPro is not as opensource as DiscFerret, but their format don't need to be hacked (they provide full specifications) and their protocol neither (also, documentation provided).
I agree with you that adding support for STREAM files is a must. But I don't think the same of their dumping hardware. STREAM files are a perfectly fine medium for lossless transportation of floppy dumps, no need to support the physical KF device. It would be nice to have, yes. But definitely not required.

It's a shame that KF is closed-source, I agree, but let's face it, it's simply the bestsolution out there for dumping floppy disk with their copy protection intact. The guys behind KF are long-time experts in the field of floppy disk technology, the single guy behind SCP is an individual who is not the easiest person around to work with (from what I heard... I don't know Jim personally so take my opinion with a grain of salt).
Funny you say that, I can personally say the same about the KF team xD
Darkstar wrote:There is probably a reason almost every institution (libraries, museums, etc.) are using a KF instead of the SCP... ;-)
That is very simple. SCP is new, KF existed for a loooooooong time.

mrpijey
User avatar
Administrator
Posts: 8183
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: [RFC] DiscImageChef

Post by mrpijey »

Which is why it's important to support the KF as well...

Regardless of personal opinions of both devices they need to be supported one way or another if the tool is supposed to deal with preservation functionality and formats.
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

mrpijey wrote:Which is why it's important to support the KF as well...

Regardless of personal opinions of both devices they need to be supported one way or another if the tool is supposed to deal with preservation functionality and formats.
Don't worry, you'll see both.

There are things much more difficult to support, like DART, XPACK, to name a few.

mrpijey
User avatar
Administrator
Posts: 8183
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: [RFC] DiscImageChef

Post by mrpijey »

Of course. But keep up the good work, but be sure you have some design goals with your tool, and think about the backup-restore functionality I mentioned before as I see is the most critical function in a backup tool.
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

mrpijey wrote:Of course. But keep up the good work, but be sure you have some design goals with your tool, and think about the backup-restore functionality I mentioned before as I see is the most critical function in a backup tool.
"Media preservation tool" is more appropriate :p

You'll have the ability to dump .MDF/.MDS from discs in next release.

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

Hi all,

I have made a pre-release available at https://github.com/claunia/DiscImageChe ... /v3.5.99.0 so people can help me testing device support under Windows.

The worst it can do to your computer is force you to reboot. It should not damage your data.
However as a pre-release I cannot guarantee it will not become conscious and want to take over the world.

Instructions on how to test on Windows: https://pastebin.com/Tgisxn0Q

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

Greetings programs

New version 4.0.0.0 released.

It is a HUGE list of changes that you can check on https://github.com/claunia/DiscImageChe ... g/v4.0.0.0

But amongst other things, dumping now generates an Alcohol 120% image (no data position measurement right now) for applicable media, fully supports dumping optical discs, magnetic disks, usb sticks, superfloppies and streaming tapes on Windows, Linux and FreeBSD.

Merry holidays to y'all.

mrpijey
User avatar
Administrator
Posts: 8183
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: [RFC] DiscImageChef

Post by mrpijey »

A great tool becomes even greater. Keep up the good work!
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

Greetings programs,

If you think things went cold, you were absolutely wrong.

Released v4.5.0.1663 at https://github.com/claunia/DiscImageChe ... 4.5.0.1663 with 60 new features, 42 fixes and 20 changes.

The most important feature, is, the ability to write quite a bunch of formats. That means you can use it to dump media to your favorite format, e.g. Alcohol 120%! (data position measurement NOT YET supported).

The next release will include a lot of changes around floppy formats, including support for bitstream floppy formats (HFE, etc), flux floppy formats (KryoFlux, SuperCardPro) and the ability to dump floppies from different controllers, so stay tuned for more goodness.

Disclaimer: Because most formats are propietary and non documented, the output from DiscImageChef may not be the same byte-by-byte as the output created by the original software. However the original software should be able to open the output generated from DiscImageChef and the contents should be the same. Would this not be the case, please open an issue in GitHub with the image as generated by DiscImageChef and the image as generated by the original software.

Have a nice summer ahead!

claunia
FTP Access
Posts: 49
Joined: Tue Aug 07, 2012 3:08 pm

Re: [RFC] DiscImageChef

Post by claunia »

After a few bug reports about generation of Alcohol 120% images with DiscImageChef I have updated to version v4.5.1.1692 that you can download at https://git.io/fNc6L

If you have created Alcohol 120% Compact Disc images with previous versions, they wouldn't be correct.

To check what to do you need to use the image-info command with them.

If they show any MODE2 tracks or are for more than 60 minutes (will show as 01:mm:ss:ff) you just need to use the convert-image command to write them to a corrected format.
However if they are multisession, or they change track mode (e.g. from Audio to MODE 1 or from MODE 2 Form 1 to MODE 2 Form 2), you'll have to dump them again.

Sorry for any inconveniences, but for the good side, now all images created by DiscImageChef can be mounted with Alcohol 120% and Daemon Tools.

P.S.: Alcohol 120% does not support CD-i Ready or CD-i without tracks formats, so if you have one of those, don't use the Alcohol 120% format. Currently the only format supporting those two types of discs correctly is dicformat (CDRWIN and CloneCD can read those discs but they lose information about their structure).

Post Reply