BetaArchive Logo
Navigation Home Database Screenshots Gallery Image Uploader Server Info FTP Servers Wiki Forum RSS Feed Rules Please Donate
UP: 12d, 23h, 5m | CPU: 50% | MEM: 5305MB of 11607MB used
{The community for beta collectors}

Post new topic Reply to topic  [ 44 posts ]  Go to page Previous  1, 2
Author Message
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Wed Aug 16, 2017 2:13 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7935
Well, one critical part about every preservation tool is being able to recreate the image onto the format it originated from, so I hope you'll add that functionality in the future. Or at least being able to convert to a format so we can use other tools.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable) :: DiscImageCreator
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Wed Aug 16, 2017 5:46 pm 
Reply with quote
FTP Access
Offline

Joined
Wed Jan 11, 2017 12:37 pm

Posts
63

Favourite OS
Whistler or Longhorn?
I agree with mrpijey.

Having your own dumping format is cool (and you should keep it... it's a bit like the Kryoflux raw one) but if your tool can -for example (and like the dtc Kryoflux tool)- create the exact same MDF/MDS file from a DVD (or from a set of raw files) than Alcohol 120%, it will be really interesting. Because it will mean your dump will be usable easily. Version 5.0 will be the interesting one.

Dumping floppy from a real 3.5" drives or 5.25" would be cool to. This way it will be able to replace many smaller tools.

Anyway, I can understand - due tot he variety of functions offered by your tool - that it targets different types of users with different needs.


Top  Profile
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Wed Aug 16, 2017 5:55 pm 
Reply with quote
FTP Access
Offline

Joined
Tue Aug 07, 2012 3:08 pm

Posts
49

Favourite OS
OS/2 Warp 4
tristanleboss wrote:
I agree with mrpijey.

Having your own dumping format is cool (and you should keep it... it's a bit like the Kryoflux raw one) but if your tool can -for example (and like the dtc Kryoflux tool)- create the exact same MDF/MDS file from a DVD (or from a set of raw files) than Alcohol 120%, it will be really interesting. Because it will mean your dump will be usable easily. Version 5.0 will be the interesting one.

Dumping floppy from a real 3.5" drives or 5.25" would be cool to. This way it will be able to replace many smaller tools.

Anyway, I can understand - due tot he variety of functions offered by your tool - that it targets different types of users with different needs.


I won't use my own dumping format, even if I designed one extensible format, because that would mean only my tool supports it.
For CDs I will use BIN/CUE and for DVDs/BDs MDF/MDS.
For other devices there is no need to do anything, it already creates IMG.

Dumping floppies from real drives can be done on three ways:
1.- Using an USB floppy drive. This is already supported, but because of the drives themselves, it's limited to non-copy-protected disks in a handful of formats.
2.- Using a PC floppy controller. This is doable, but don't expect it anytime soon.
3.- Using a kryoflux/discferret/supercardpro. This is huge problem. KryoFlux is secretive, everything but what conservation means, yet it's the most common hardware, so I would need to reverse engineer their USB protocol (and this, takes a LOT of time). DiscFerret and SuperCardPro should be easier to do, their not as secretive, but I don't have the hardware to test it, and without it, I just wouldn't be able to support them.

In a perfect world, where everything goes as planned, you'll see these come alive:
BIN/CUE and MDF/MDS in DiscImageChef 4.0 (planned in a few months)
Image format conversion in DiscImageChef 5.0 (Xmas '17)
Supporting kryoflux images in DiscImageChef 5.5 (2018)
PC floppy controller in DiscImageChef 5.5 (2018)
Creating images from a kryoflux device in DiscImageChef 6.0 (2019 or 2020)
Creating images from a discferret or supercardpro device in DiscImageChef 5.5 if someone donates me the controllers, if not, probably never for discferret (as it's no longer manufactured) and who knows when for supercardpro.


Top  Profile
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Thu Aug 17, 2017 11:27 pm 
Reply with quote
Donator
User avatar
Offline

Joined
Fri May 14, 2010 1:29 pm

Posts
865

Location
Southern Germany

Favourite OS
IRIX 5.3
claunia wrote:
3.- Using a kryoflux/discferret/supercardpro. This is huge problem. KryoFlux is secretive, everything but what conservation means, yet it's the most common hardware, so I would need to reverse engineer their USB protocol (and this, takes a LOT of time). DiscFerret and SuperCardPro should be easier to do, their not as secretive, but I don't have the hardware to test it, and without it, I just wouldn't be able to support them.

The KryoFlux USB protocol is probably a very simple one. I'd suggest running Wireshark on your USB port while dumping a disk or two, and seeing what you get. My guess it's that it's rather easy to replicate. But I might be wrong.

If you want, I can provide you with some USB traces and the corresponding command line / raw dumps for a handful of floppies, but at the moment I don't have enough free time to tinker with reverse engineering the protocol myself, sadly

_________________
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA


Top  Profile
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Fri Aug 18, 2017 12:00 am 
Reply with quote
FTP Access
Offline

Joined
Tue Aug 07, 2012 3:08 pm

Posts
49

Favourite OS
OS/2 Warp 4
Darkstar wrote:
claunia wrote:
3.- Using a kryoflux/discferret/supercardpro. This is huge problem. KryoFlux is secretive, everything but what conservation means, yet it's the most common hardware, so I would need to reverse engineer their USB protocol (and this, takes a LOT of time). DiscFerret and SuperCardPro should be easier to do, their not as secretive, but I don't have the hardware to test it, and without it, I just wouldn't be able to support them.

The KryoFlux USB protocol is probably a very simple one. I'd suggest running Wireshark on your USB port while dumping a disk or two, and seeing what you get. My guess it's that it's rather easy to replicate. But I might be wrong.

If you want, I can provide you with some USB traces and the corresponding command line / raw dumps for a handful of floppies, but at the moment I don't have enough free time to tinker with reverse engineering the protocol myself, sadly


Darkstar I already stored one USB trace. I don't think there's need for more than one, it's streaming, dtc just asks kryoflux for data and everything is streamed over usb. Nonetheless if you create more traces, better yet.
The problem is I don't have the time myself either :/

Maybe there's people interested on paying our time on a bounty? I do not move in those forums but from what I've heard, lots of people want to be able to use the kryoflux with a raspberry pi, something their creators have repeatedly said they won't support, but DiscImageChef will have no problem doing.


Top  Profile
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Fri Aug 18, 2017 1:55 am 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7935
I think we should leave the actual dumping to the KF team as there's more to it than just streaming USB data... but conversion and reading support could be implemented using their libraries.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable) :: DiscImageCreator
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Fri Aug 18, 2017 2:51 am 
Reply with quote
FTP Access
Offline

Joined
Tue Aug 07, 2012 3:08 pm

Posts
49

Favourite OS
OS/2 Warp 4
mrpijey wrote:
I think we should leave the actual dumping to the KF team as there's more to it than just streaming USB data... but conversion and reading support could be implemented using their libraries.


There's a problem, their libraries.

See, KryoFlux just streams the pulses from the drive and the internal clock thru USB. They even call their output STREAM, that they don't document.
Their software, aka dtc, stores it, and for some formats, creates a RAW image, that, btw loses all tags for Lisa and MacOS double density disks (MFS filesystem uses the tags for filesystem recovery, LisaOS requires it for reading the disk). A lot of disks (XDF, copy protected) just don't work.
Then they want you to send them the STREAM to create, yet another closed source undocumented format called IPF using a software that they sell under NDA (so far for software preservation). Originally they provided only binary blobs, later they provided source code under a license that is not opensource, neither OSI neither GNU compliant. You can read it here: https://github.com/lunixbochs/fs-uae-gl ... ge/LICENSE
This by itself prevents using their libraries (even looking at the code in that folder may damage DiscImageChef development).

That's why people like Keirf (https://github.com/keirf/Disk-Utilities/) and Jean Louis (http://info-coach.fr/atari/software/preservation.php) created their own utilities to convert the STREAM files to appropriate files, being RAW or files that support copy protection for specific emulators.

Adding support for KryoFlux STREAM files, and the device itself, is a must, because it's so extended, people is creating the need to depend on a group that's closed, secretive, and everything but preserving.
Also it's by far the more closed, least powerful, and least recommended of the equivalent devices.

DiscFerret is fully opensource, and powerful enough to dump even hard disks, but unfortunately abandoned by its creator it cannot be bought anymore.
SuperCardPro is not as opensource as DiscFerret, but their format don't need to be hacked (they provide full specifications) and their protocol neither (also, documentation provided).

More open -> more propietary
DiscFerret -> SuperCardPro -> KryoFlux


Top  Profile
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Fri Aug 18, 2017 6:57 am 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7935
And yet it's the first hit I found when I looked for it...

https://www.kryoflux.com/download/kryof ... rev1.1.pdf

Oh look, another one for IPF

http://info-coach.fr/atari/documents/_m ... tation.pdf

Sure the second one is not directly from source devs, but looks pretty well documented to me. Quite a few tools and emulators support it as well, so it must be possible to implement support for it one way or another...

I do however agree on SPS's closed "preservation" tactics which are anything but preservation, which is why I stopped donating stuff to them. No point of sharing stuff if the point of failure can be traced down to a single place...

KryoFlux seems to be the more used and preferred tool as well, a lot due to the better hardware and more open and active community around it. I never had issued with it, written back a lot of floppies from the STREAM format without any issues whatsoever, and the software is just better... (yes I own an SCP as well, and its software is dodgy at best).

What I am saying is that your tool doesn't need to support the hardware devices directly, only the formats so it can convert from them (if you intend your tool to also be able to read from one format and write to another, if not then ignore everything I've said).

Edit: But the main question comes down to: What are your intentions with your software? If you want to have preservation quality features in it then it need to

  1. support other formats such as STREAM, IPF, SCP and other low level formats, at least be able to read from them and convert.
  2. support full read and write of your own format if you choose to use one of your own
  3. be fully documented, at least to the extent that others can use your own format.
  4. support the basic storage hardware, such as floppy drives and (S/P)ATA optical drives.
  5. read the media at its lowest possible level to preserve as much info as possible.

If you don't intend it to be a tool for preservation but just one to read various storage media then fine, but then it's not suitable for preservation either and should not be used as such. There are already so many tools out there that handles the various formats, but not one that does all of it well. Considered to make it modular? Allow a plugin system where others can develop their own plugins to support all kinds of stuff and then lift some of the pressure off you.

But the important thing is that you can read and write your own format. A tool that can't use its own format to recreate the original is pretty much useless, since what use do you have of a bunch of image files if they are unusable? It's like storing a picture in a format no other applications can handle... And if you decide to only support the high level stuff such as ISO, BIN/CUE etc then that's fine, but for BetaArchives preservation purposes it won't be enough.

Don't get me wrong, I am very interested in your software and it looks like a very nice candidate for an all-in-one media dumper, but it has to be able to preserve as much info as possible of a media, and do it in a way that can either be recreated (using your software) or dumped into a format other low level tools can handle, such as KryoFlux RAW/STREAM, MDF/MDS/MDX, CloneCD etc, so it can be mountable and usable outside your tool.

Keep up the good work!

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable) :: DiscImageCreator
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Fri Aug 18, 2017 9:34 am 
Reply with quote
Donator
User avatar
Offline

Joined
Fri May 14, 2010 1:29 pm

Posts
865

Location
Southern Germany

Favourite OS
IRIX 5.3
claunia wrote:
Adding support for KryoFlux STREAM files, and the device itself, is a must, because it's so extended, people is creating the need to depend on a group that's closed, secretive, and everything but preserving.
Also it's by far the more closed, least powerful, and least recommended of the equivalent devices.

DiscFerret is fully opensource, and powerful enough to dump even hard disks, but unfortunately abandoned by its creator it cannot be bought anymore.
SuperCardPro is not as opensource as DiscFerret, but their format don't need to be hacked (they provide full specifications) and their protocol neither (also, documentation provided).


I agree with you that adding support for STREAM files is a must. But I don't think the same of their dumping hardware. STREAM files are a perfectly fine medium for lossless transportation of floppy dumps, no need to support the physical KF device. It would be nice to have, yes. But definitely not required.

It's a shame that KF is closed-source, I agree, but let's face it, it's simply the bestsolution out there for dumping floppy disk with their copy protection intact. The guys behind KF are long-time experts in the field of floppy disk technology, the single guy behind SCP is an individual who is not the easiest person around to work with (from what I heard... I don't know Jim personally so take my opinion with a grain of salt).

There is probably a reason almost every institution (libraries, museums, etc.) are using a KF instead of the SCP... ;-)

_________________
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA


Top  Profile
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Fri Aug 18, 2017 1:37 pm 
Reply with quote
FTP Access
Offline

Joined
Tue Aug 07, 2012 3:08 pm

Posts
49

Favourite OS
OS/2 Warp 4
Darkstar wrote:
claunia wrote:
Adding support for KryoFlux STREAM files, and the device itself, is a must, because it's so extended, people is creating the need to depend on a group that's closed, secretive, and everything but preserving.
Also it's by far the more closed, least powerful, and least recommended of the equivalent devices.

DiscFerret is fully opensource, and powerful enough to dump even hard disks, but unfortunately abandoned by its creator it cannot be bought anymore.
SuperCardPro is not as opensource as DiscFerret, but their format don't need to be hacked (they provide full specifications) and their protocol neither (also, documentation provided).


I agree with you that adding support for STREAM files is a must. But I don't think the same of their dumping hardware. STREAM files are a perfectly fine medium for lossless transportation of floppy dumps, no need to support the physical KF device. It would be nice to have, yes. But definitely not required.

It's a shame that KF is closed-source, I agree, but let's face it, it's simply the bestsolution out there for dumping floppy disk with their copy protection intact. The guys behind KF are long-time experts in the field of floppy disk technology, the single guy behind SCP is an individual who is not the easiest person around to work with (from what I heard... I don't know Jim personally so take my opinion with a grain of salt).

Funny you say that, I can personally say the same about the KF team xD

Darkstar wrote:
There is probably a reason almost every institution (libraries, museums, etc.) are using a KF instead of the SCP... ;-)

That is very simple. SCP is new, KF existed for a loooooooong time.


Top  Profile
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Fri Aug 18, 2017 2:00 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7935
Which is why it's important to support the KF as well...

Regardless of personal opinions of both devices they need to be supported one way or another if the tool is supposed to deal with preservation functionality and formats.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable) :: DiscImageCreator
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Fri Aug 18, 2017 2:02 pm 
Reply with quote
FTP Access
Offline

Joined
Tue Aug 07, 2012 3:08 pm

Posts
49

Favourite OS
OS/2 Warp 4
mrpijey wrote:
Which is why it's important to support the KF as well...

Regardless of personal opinions of both devices they need to be supported one way or another if the tool is supposed to deal with preservation functionality and formats.


Don't worry, you'll see both.

There are things much more difficult to support, like DART, XPACK, to name a few.


Top  Profile
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Fri Aug 18, 2017 2:16 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7935
Of course. But keep up the good work, but be sure you have some design goals with your tool, and think about the backup-restore functionality I mentioned before as I see is the most critical function in a backup tool.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable) :: DiscImageCreator
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Fri Aug 18, 2017 2:21 pm 
Reply with quote
FTP Access
Offline

Joined
Tue Aug 07, 2012 3:08 pm

Posts
49

Favourite OS
OS/2 Warp 4
mrpijey wrote:
Of course. But keep up the good work, but be sure you have some design goals with your tool, and think about the backup-restore functionality I mentioned before as I see is the most critical function in a backup tool.

"Media preservation tool" is more appropriate :p

You'll have the ability to dump .MDF/.MDS from discs in next release.


Top  Profile
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Tue Sep 05, 2017 5:53 pm 
Reply with quote
FTP Access
Offline

Joined
Tue Aug 07, 2012 3:08 pm

Posts
49

Favourite OS
OS/2 Warp 4
Hi all,

I have made a pre-release available at https://github.com/claunia/DiscImageChe ... /v3.5.99.0 so people can help me testing device support under Windows.

The worst it can do to your computer is force you to reboot. It should not damage your data.
However as a pre-release I cannot guarantee it will not become conscious and want to take over the world.

Instructions on how to test on Windows: https://pastebin.com/Tgisxn0Q


Top  Profile
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Sun Dec 24, 2017 9:30 pm 
Reply with quote
FTP Access
Offline

Joined
Tue Aug 07, 2012 3:08 pm

Posts
49

Favourite OS
OS/2 Warp 4
Greetings programs

New version 4.0.0.0 released.

It is a HUGE list of changes that you can check on https://github.com/claunia/DiscImageChe ... g/v4.0.0.0

But amongst other things, dumping now generates an Alcohol 120% image (no data position measurement right now) for applicable media, fully supports dumping optical discs, magnetic disks, usb sticks, superfloppies and streaming tapes on Windows, Linux and FreeBSD.

Merry holidays to y'all.


Top  Profile
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Sun Dec 24, 2017 10:56 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7935
A great tool becomes even greater. Keep up the good work!

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable) :: DiscImageCreator
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Sun Jun 24, 2018 1:26 pm 
Reply with quote
FTP Access
Offline

Joined
Tue Aug 07, 2012 3:08 pm

Posts
49

Favourite OS
OS/2 Warp 4
Greetings programs,

If you think things went cold, you were absolutely wrong.

Released v4.5.0.1663 at https://github.com/claunia/DiscImageChe ... 4.5.0.1663 with 60 new features, 42 fixes and 20 changes.

The most important feature, is, the ability to write quite a bunch of formats. That means you can use it to dump media to your favorite format, e.g. Alcohol 120%! (data position measurement NOT YET supported).

The next release will include a lot of changes around floppy formats, including support for bitstream floppy formats (HFE, etc), flux floppy formats (KryoFlux, SuperCardPro) and the ability to dump floppies from different controllers, so stay tuned for more goodness.

Disclaimer: Because most formats are propietary and non documented, the output from DiscImageChef may not be the same byte-by-byte as the output created by the original software. However the original software should be able to open the output generated from DiscImageChef and the contents should be the same. Would this not be the case, please open an issue in GitHub with the image as generated by DiscImageChef and the image as generated by the original software.

Have a nice summer ahead!


Top  Profile
 PostPost subject: Re: [RFC] DiscImageChef        Posted: Thu Jul 19, 2018 11:37 pm 
Reply with quote
FTP Access
Offline

Joined
Tue Aug 07, 2012 3:08 pm

Posts
49

Favourite OS
OS/2 Warp 4
After a few bug reports about generation of Alcohol 120% images with DiscImageChef I have updated to version v4.5.1.1692 that you can download at https://git.io/fNc6L

If you have created Alcohol 120% Compact Disc images with previous versions, they wouldn't be correct.

To check what to do you need to use the image-info command with them.

If they show any MODE2 tracks or are for more than 60 minutes (will show as 01:mm:ss:ff) you just need to use the convert-image command to write them to a corrected format.
However if they are multisession, or they change track mode (e.g. from Audio to MODE 1 or from MODE 2 Form 1 to MODE 2 Form 2), you'll have to dump them again.

Sorry for any inconveniences, but for the good side, now all images created by DiscImageChef can be mounted with Alcohol 120% and Daemon Tools.

P.S.: Alcohol 120% does not support CD-i Ready or CD-i without tracks formats, so if you have one of those, don't use the Alcohol 120% format. Currently the only format supporting those two types of discs correctly is dicformat (CDRWIN and CloneCD can read those discs but they lose information about their structure).


Top  Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 44 posts ]  Go to page Previous  1, 2




Who is online

Users browsing this forum: No registered users and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  

All views expressed in these forums are those of the author and do not necessarily represent the views of the BetaArchive site owner.

Powered by phpBB® Forum Software © phpBB Group

Copyright © 2006-2019

 

Sitemap | XML | RSS