Disk Copy 4.2, and etc.

[lilliputian]

Hello,

My name is Lilliputian and I am a regular contributor to the Macintosh Garden. I just discovered BetaArchive and am interested in applying for FTP access, but I wanted to verify the procedure before submitting my application.

I have read through the guidelines, but very little is said about formatting prior to OS X. In particular, much of the software I deal with is stored on Macintosh 800k (or 400k) floppies. From older posts on the forum, I have gathered that Disk Copy 4.2-format disk images are preferred, though it is not mentioned in the guidelines. This is necessary, since they can't be dd'd on modern floppy drives like 1.44MB diskettes. DC42 is also the preference on the Garden for such disks, since Disk Copy 6.x doesn't preserve everything correctly, even when saving in DC42 format.

However, the other element is the resource fork. DC42 doesn't store any of its imaged data in the resource fork, but the File Type and Creator Codes, used by Classic Mac OS to identify files, are. As many will no doubt know, when non-Macintosh file systems get ahold of Classic Mac files, the resource forks are instantly stripped away. Not a problem for a JPEG, but this renders many Macintosh files useless (in particular Application programs). Something like a DC42 disk image can have its resource fork and type codes restored using ResEdit (or similar), but this is an extra step that can be avoided by using binary encoding first. Way back in time, this involved UUEncode, or a program called PackIt, but later was superseded by BinHex, specifically BinHex 4 (.hqx) and the later MacBinary format (.bin, aka BinHex 5). By encoding the file's resource fork in the data fork (basically converting the file into plain text), the resource fork can be preserved on non-Macintosh file systems, and only requires de-encoding by the user upon downloading before opening.

So the question is, which, if any, encoding is preferred here?

On the Garden, there is no strict preference, it's left up to the uploader. I personally use .hqx, as it was the most common in my experience prior to OS X, but I'm open to other formats. MacBinary is technically smaller.

Also, is it acceptable to compress multiple images into a StuffIt archive prior to encoding (I use v1.5.1 for compatibility), or should each disk image be encoded separately? (BinHexing only works on individual files, not multiple.)

Also, also, the default dc42 file extension of ".image" is often truncated afterwards to ".img" by the user imaging the disk (myself included); would it be preferred though to leave it as ".image" for BetaArchive? (File extensions don't generally mean anything to the system in Classic Mac OS prior to X, they're just there for the user's convenience).

Thank you! Looking forward to contributing.

[mrpijey]

When it comes to classic mac we don't have any specific preference, but we do want a format that doesn't rely on the HFS+ resource forks. So any proper raw format of the floppy image packed up in a sit etc would be OK. So binhex etc should be fine for this.

And please do encode individual floppy images as we archive each individual floppy separately to make sure users can grab the floppy they want without the need to download everyting.

.img will be fine.

[vbdasc]

DC42 is also the preference on the Garden for such disks, since Disk Copy 6.x doesn't preserve everything correctly, even when saving in DC42 format.

Pardon me for the interruption, but can you elaborate on this?

[lilliputian]

My understanding is that not all of the disk metadata is preserved when using Disk Copy 6.x (usually 6.3.3), which is of course important for this kind of archival work. Also, according to the documentation for the FloppyEmu (a floppy emulator for classic Macs, the Lisa, and the Apple II from BMOW), apparently DiskCopy 6.x relies on the Macintosh resource fork in a way that Disk Copy 4.2 does not. (There was no official release of Disk Copy 5, if you were wondering.)

However, Disk Copy 6 can still play a role, as it is capable of actually mounting disk images on the Desktop, whereas Disk Copy 4.2 is not. (Older systems prior to System 7 which can't run Disk Copy 6 can instead use the control panel "MountImage".)

[vbdasc]

Yes, I see. Now I'm going to engage in even more off-topic, first because I would like to dispel the raised doubts about the use of DiskCopy 6.3.3 (and DiskCopy 6.x in general), and second, because I like to show my knowledge (okay, this was a joke, and my knowledge is not large at all and is rather patchy, but I just happen to know something about DiskCopy images)

I read the FloppyEmu documentation and it indeed seems to have some problems with DiskCopy 6.x. I will try to list them

first, it says that "DiskCopy 6.x format" is not common. This is okay, but what is "DiskCopy 6.x format"? AFAIK, DiskCopy 6.x supports 4 different image sub-formats: "Read-write", "Read-only," "Compressed" and "DC4.2 compatible". Which one?

second, it says that it needs both forks for integrity, and both forks are difficult to be supported in a non-Mac filesystem. But according to my tests, only the "Compressed" sub-format actually needs its resource fork. "Read-write" data fork is just the raw sector image, "Read-only" data fork is the same, but with truncated last sectors which are unused by the file system, and "DC4.2 compatible" is just that, the same format as in DiskCopy 4.2 . And I believe that a raw sector image is just perfect for archiving a disc. Speaking about the disc metadata, I don't think there's any important. Correct me if I'm wrong. About the "Compressed" sub-format - yes, if you lose the resource fork, the data fork alone is useless. Just don't use "Compressed". Problem solved.

third, the Mac OS X image utility hdiutil has a bug and corrupts DiskCopy 6.x images. Solution: Don't use hdiutil.

Conclusion: there are no actual problems with DiskCopy 6.x images. I'm not sure about the disc metadata it fails to preserve, but anyway, as I said, any disc metadata is probably not important at all when archiving a diskette.

[lilliputian]

Disk Copy 6 introduced its own format called NDIF ("New Disk Image Format"). Additionally, while it is possible to create an NDIF image without compression, the default when making them is compressed. If you save as dc42 compatible, it just saves it as a dc42 image, I believe. However, it will lack the aforementioned metadata.

This metadata includes the create/modification dates of the disk itself, I believe, which seems important to me. Additionally, the older systems that can't run Disk Copy 6 (pre-System 7) are better served by having the images in the dc42 format they can actually use, so you may as well just use Disk Copy 4.2.

Regarding hdiutil, I didn't know that myself. I'm unlikely to have gone that route in any case, but good to know!

[DOS]

Disk Copy 6 introduced its own format called NDIF ("New Disk Image Format"). Additionally, while it is possible to create an NDIF image without compression, the default when making them is compressed. If you save as dc42 compatible, it just saves it as a dc42 image, I believe. However, it will lack the aforementioned metadata.

This metadata includes the create/modification dates of the disk itself, I believe, which seems important to me.

Hi, welcome to the forums!

When metadata was mentioned earlier, I imagined things like a disk manufacturer name, model number and serial number as you might extract from a hard disk drive, i.e. information about the disk which isn't in the disk's data area. For floppies, I would speculate that there isn't anything similar that anyone would care about. I don't for example recall hearing that there is a special sector on a floppy disk which contains creation/modification dates which aren't read by a normal read of all of the sectors. I suppose you can obtain information the manufacturer from the drive, but it would be the manufacturer of the drive and not the disk, which doesn't seem all that important unless someone wants to know how the image was created? Creation/modification dates for the disk itself sounds like something that would be stored in the filesystem and hence end up in the raw disk image. When the image was created sounds like metadata people would be interested in, but that's metadata which our host filesystems already deal with.

I could be mistaken about all of that though, and in any case I don't have any preference for Disk Copy 6!

[lilliputian]

To be perfectly honest, it isn't my expertise, but it's been discussed on the Macintosh Garden in the past, and the official recommendation there is Disk Copy 4.2. Given that 4.2 runs on systems prior to System 7 (and DC6 does not), it's easiest to stick with that except when you can't (DC4.2 stops working in late versions of classic Mac OS, although I'm not sure where the cutoff is).

Also I was mistaken: Disk Copy 6 does not seem to default to compressed images. However, it is still the later NDIF format, which is not a raw image.

(Technically, neither is the dc42 format, bit it is only differentiated by the addition of a header I believe, and if you really want to, there is a utility by BMOW that can convert dc42 images to true raw dsk format.)

[Hyoenmadan86]

From https://www.bigmessowires.com/2015/02/2 ... emulation/

"On both the Lisa and the Macintosh, a floppy sector’s data contains 12 bytes of tags, plus 512 bytes of normal data. On the Mac, the tags aren’t used for anything, and Floppy Emu just treats them as if they were always zero. Standard .dsk image files (dd and similars) don’t even store the tag bytes, although Disk Copy 4.2 images do. But unlike the Mac, the Lisa needs those tag bytes in order for Lisa-formatted disk images to work correctly. So at a minimum, the firmware will need to fetch the tag bytes for each sector from the DC42 image file, instead of just pretending they’re zero."

In resume, from preservation standpoint you would like to have EVERY data from the disk sectors as proof the dump is genuine and can be exactly reconstructed if needed in its original state, which only DC42 images can do. Many preservation archives will demand DC42 only images (dunno about BetaArchive policies regards Mac software diskette archiving, specially when you want to upload a piece in order to get FTP access to the site archive) due its ability to retain all the floppy sector data [12bytes tag + 512bytes normal data], which can't be done with any other version of Diskcopy (Discopy 6 "DC42" images are just DSK images) or 3rd party tools. From technical standpoint, is pointless unless you are imaging Lisa diskettes.

[lilliputian]

Thank you for adding the info about tags. Macintosh 800k and 400k diskettes utilize "tags" which contain information about the files written to disk. These were only used for these size diskettes (either MFS or HFS) and not for 1.4MB diskettes, nor for any hard drives or mass storage (zip disks, bernoulli, syquest, etc). So thank you for confirming that that is indeed the info that is preserved in the DC4.2 format!

Indeed, when imaging an 800k or 400k diskette using DC4.2, it reports two separate checksums afterwards: one for the primary data, and one just for the tags. And while the tags are perhaps not necessary for the functioning of the diskettes on the Macintosh, they can be used in some cases for file recovery, should a disk be damaged. Though still perhaps not strictly necessary for archival efforts.

But, while DiskCopy 6 itself can't be used pre-System 7, neither can the NDIF images it produces. So, once again, as just a matter of practicality, it probably makes most sense to just use DC4.2 in the first place (or something like DiskDup, which can save in that format, including tag info, and is much more tolerant than Disk Copy when it comes to read errors).

[vbdasc]

In resume, from preservation standpoint you would like to have EVERY data from the disk sectors as proof the dump is genuine and can be exactly reconstructed if needed in its original state, which only DC42 images can do.

You seem to be right; I only have a couple of remarks. First, there is some kind of confusion on the Web about the number of tag bytes. Some sources say that they're 20 per sector. (P.S. Please ignore the above remark. It's quite certain that Mac/Lisa 400K/800K diskettes only have 12 bytes of tag data per sector. It seems that the "Twiggy" diskettes for Lisa use 20 tag bytes, but they're another topic. My mistake.) Second, they only exist on 400K diskettes (and possibly on 800K too), but not on 1.44M ones (which are accessed by a PC-like NEC_uPD765-compatible floppy controller, so there is just no place for them). Yes, it is desirable to archive all software-accessible bytes and bits on a diskette, plus from a technical standpoint, if a regular application like DC 4.2 can access the tag bytes, then they can be usable as a form of copy-protection, and this makes preserving them absolutely necessary.

A floppy diskette actually contains much more data then what is contained in its sectors, though; and not all of it can even be expressed easily in binary form; this is why things like Kryoflux exist! The key and the answer where to draw the line where we stop putting data into images for archiving should be the software accessibility IMHO. Tag bytes were software accessible, so their place is in the images. More low-level data is not accessible, at least not without ugly hacks, so its place is not in the images.

Macintosh 800k and 400k diskettes utilize "tags" which contain information about the files written to disk.

What do you mean? It's clear enough that the tags weren't assigned any special meaning under any Mac filesystem. Both the MFS and HFS filesystems operated strictly on 512-byte sectors and didn't contain by themselves any "mirroring" features to the tag bytes. There may have been Mac utilities which have been doing exactly that, but none of this was standardized.

[lilliputian]

Perhaps I misunderstand, but if a utility can use the tags on disk to help recover data (specifically, I am thinking of a program called "1st Aid HFS" which can do this), then they must contain some information about the files on the disk, no? Naturally, they are not a duplicate or mirror of the files themselves.

[DOS]

Thanks very much for the clarifications and interesting details!

[vbdasc]

Perhaps I misunderstand, but if a utility can use the tags on disk to help recover data (specifically, I am thinking of a program called "1st Aid HFS" which can do this), then they must contain some information about the files on the disk, no? Naturally, they are not a duplicate or mirror of the files themselves.

Yes, but the point is that the tag bytes on a 400K/800K MFS/HFS formatted Mac floppy are non-standardized, and may (or not) contain actual useful data. You can't rely on them for anything. I would rewrite your statement from above as:

Macintosh 800k and 400k diskettes utilize "tags" which can contain information about the files written to disk (or about anything else, if at all).

By the way, are you aware of the MOOF format (http://fileformats.archiveteam.org/wiki/MOOF)? I recently encountered a bunch of floppy images of games and other software in this format. Allegedly, it works on lower level and catches most tricks used to make copy-protected diskettes, unlike simpler formats like DC42. It seems to be the most promising format for archiving copy-protected Mac diskettes. The downside is that barely any software supports it (but recent MAME versions apparently do).

[lilliputian]

I have heard of MOOF. From what I understand, any of these low-level magnetic flux readers (like also the KryoFlux) seem to be able to copy disks with certain protection schemes, like ones with seemingly random bits, special bits outside the data area, and etc, since they're not actually reading or interpreting any data, just the magnetic flux itself. And of course they can also be helpful with data recovery when nothing else will work. I have never tried any of them myself, however.

I would be interested, but I don't have a pressing need for one currently, and they are a bit expensive.

EDIT: And for the purposes of most software archives, it seems like a bit overkill, and more difficult to use as actual disk images (apart from getting around copyprotection, etc).