ISO or ESD?

[mrpijey]

I am starting this poll because I want to get the community input on a thought I've had...

I have been thinking about the usability of all the ESD files Microsoft are pushing out. Apart from the betas we already got I've managed to collect about 6TB of additional ESD files from post-RTM Win10 builds and this collection grows by the month. By the time Microsoft ends the development cycle of Windows 10 and it becomes abandonware we'll most likely have an ESD collection that will surpass the entire BA FTP collection by a factor of 10... Add to this also Server 2016 ESDs that Microsoft will push out along with more Win10 updates. This is extremely taxing on space because ESD files don't deduplicate at all, and they require post-processing before you can use them for anything decent. I do realise that ESDs are the original files and by default we try to keep them, but out of cost as well as management considerations we might make an exception. Along with each replaced ESD we would still include full file information about the original ESD (filename, checksum hashes etc) just to keep a reference of where it came from.

So my question is simple: Should we keep the ESDs as-is, or should we replace them with ISOs?

Keeping both is not an option as it would increase space requirements even further.

Here are the results of each option...


Results from keeping ESDs as-is:

• Original from Microsoft
• Slightly smaller
• Does not deduplicate at all (greatly increases current and future storage needs)
• Not usable at all until decrypted and converted

Results from replacing ESDs with (converted) ISOs:

• Slightly larger archives
• Instantly usable for USB installations, virtual machines and disc burning.
• Decreases storage needs (but increases archive size)
• Deduplicates very well (decreases storage needs)

My greatest concern at the moment is keeping down the storage needs, storage means a lot of money (need to replace SAN, controller cards and harddrives as well as backups), which is why I bring this topic up. So what should we do? Keep ESDs or replace them? I am going to make some tests and convert a build or two and see how well they deduplicate, but knowing from the Win10 betas they deduplicate extremely well as the difference between languages and SKUs are minor.

I'll leave this vote open for a month before I make a final decision.

[JimOlive]

I like having ISO's because I think they are much easier to use.

[johnleakedfan]

Esd's take less space for each file, and it is more original
But The ISO is easier to use as I agree with JimOlive, so I'm not too sure, ill just go with ESD's I guess

[longhornwhistler]

ISOs are supported by VMWare. ESDs are not last I knew.

[johnleakedfan]

ISOs are supported by VMWare. ESDs are not last I knew.

It might seem kind of obvious that A non image file wouldn't work in VMWare

[nazguz]

I've managed to collect about 6TB of additional ESD files from post-RTM Win10 builds

this is your problem
not every build or update or whatever should be archived

also multi language os-es are also waste of space

stick to english which is international an decide which builds are more unique and significant

[johnleakedfan]

I've managed to collect about 6TB of additional ESD files from post-RTM Win10 builds

this is your problem
not every build or update or whatever should be archived

also multi language os-es are also waste of space

stick to English which is international an decide which builds are more unique and significant

Alot of people here don't speak Perfect English, so is why BA uploads all these other languages of software, either it's currently available only in English or in another Language (Or even Both).

BA wouldn't have just English, as we preserve ALL software we can

[JimOlive]

I've managed to collect about 6TB of additional ESD files from post-RTM Win10 builds

this is your problem
not every build or update or whatever should be archived

also multi language os-es are also waste of space

stick to english which is international an decide which builds are more unique and significant

The purpose of BA is to archive literally everything as per the rules. Not everybody on Earth speaks english. While I agree everybody should speak english, for some reason they don't.

Going on topic, ISO's are infinitely easier to use than "ESD"s and everybody can use an ISO. People lucky enough to be allowed to actually use the FTP should get files they can actually use.

[dw5304]

while i realize you might be loosing the orignal esd when it comes to disk space i will need to agree with mrpijey converting to iso will mean alot better disk compression and thus save ba storage space.

[Hackerpcs]

Keep the ESDs, put the non-important ESDs on cold storage and have available a list available ESDs for anyone that want a specific build.

Storage cost is the only important factor in my opinion, when archiving is the aim usability is irrelevant if the process to use it is not THAT long.

[nazguz]

there is no "we", there is Andy and mpy
its their storage, unless you gonna buy them new disks of XX TB
then its up to them to choose "lesser evil"

[Zheng He]

Non-original files are useless.
If there are original ISOs, then replace ESDs with them. If you want you can keep both ESDs and original ISOs. Non-original files are always bad.

[JimOlive]

Non-original files are useless.
If there are original ISOs, then replace ESDs with them. If you want you can keep both ESDs and original ISOs. Non-original files are always bad.

Why are non-original files always bad? Non-original files can be very useful, especially if you literally can't use the ESD! It's like a picture; The original picture might be okay, but if you edit it you can make it much better! They're taking ESD files and turning them into much better ISOs.

[DiskingRound]

I think the ESDs should be kept. It's not that hard to use the ESD converter or gus33000's version to convert them to usable ISOs. And even if the decryption key isn't available (a la 10047 ESDs), the ESD should still be preserved in the event a working key is found.

[MSUser2013]

I can't decide, There's one downside about keeping them as ESDs that I can think of; The tool you need for converting them to ISOs could become extinct to the point where it's not downloadable anymore. So my suggestion to prevent that would be to upload the ESD Converter tool to the FTP in the tools folder so that way you can keep the ESDs without worrying about them being unusable in the future.

[CgSYannick]

Storage (and cost) is the more important part for me, and ISO deduplicates better. So I vote "keep ISO", even if it's not "original from Microsoft".

[mrpijey]

Thanks for all the input so far!

ISOs are supported by VMWare. ESDs are not last I knew.

Well that's because ESDs are not disc images. They don't even have the basic structure to work as such, which is why they will never be supported as such.

this is your problem
not every build or update or whatever should be archived

also multi language os-es are also waste of space

stick to english which is international an decide which builds are more unique and significant

I have to disagree here. We have an aim to save all languages as long as they are original. As many members pointed out, English is not the only language, and English speaking worlds are not the only source for computer software either. At one point we collected only english titles, but we quickly abandoned that when we realised that it goes against our preservation goals.

Keep the ESDs, put the non-important ESDs on cold storage and have available a list available ESDs for anyone that want a specific build.

Storage cost is the only important factor in my opinion, when archiving is the aim usability is irrelevant if the process to use it is not THAT long.

And who decides what's non-important? And if we put non-important ESDs on cold storage then we're back to the current problem as well, storage costs. Cold storage is just as expensive, if not even more expensive, especially the initial cost. (Anyone want to donate an Ultrium tape drive allowing for a minimum of 1TB (uncompressed) tapes? No, FYI these costs some 3000+ USD). Also, our goal is to have everything online and accessible for our members.

I think the ESDs should be kept. It's not that hard to use the ESD converter or gus33000's version to convert them to usable ISOs. And even if the decryption key isn't available (a la 10047 ESDs), the ESD should still be preserved in the event a working key is found.

Well, naturally, for the ESDs that can't be decrypted the original ESD would be kept (as we can't convert it).

I can't decide, There's one downside about keeping them as ESDs that I can think of; The tool you need for converting them to ISOs could become extinct to the point where it's not downloadable anymore. So my suggestion to prevent that would be to upload the ESD Converter tool to the FTP in the tools folder so that way you can keep the ESDs without worrying about them being unusable in the future.

Not likely as we can easily put the tools in our Tools collection (have to clear this with gus33000 first) and the fact that we got a developer for it as a member here too. The only risk is when the day comes when Microsoft abandons or changes the underlying support for ESDs, but at that point we can still virtualize a supporting OS. And when that happens ISOs will most likely still be supported as it's an universal format and will most likely stay as such as long as we got a binary computer architecture.

Well, lots of opinions and thoughts, great. At the moment it's an even race which makes it even more difficult. And I won't put in my own vote either, but as nazguz pointed out, at this time all the cost lies on me (Andy takes care of the server costs at the moment) and I can't endlessly shovel out bucketloads of money on storage. I am already planning a huge server upgrade for BA (two 8-drive DAS units as well as new controllers) and that will still not help me with increasing the storage, only make sure I can shove in enough harddrives. Just to keep up as-is I would need at least 4x4TB drives now, or 2x6TB to make sure we're ahead of the increased storage needs. And that comes out of my pocket.

[Hackerpcs]

Keep the ESDs, put the non-important ESDs on cold storage and have available a list available ESDs for anyone that want a specific build.

Storage cost is the only important factor in my opinion, when archiving is the aim usability is irrelevant if the process to use it is not THAT long.

And who decides what's non-important? And if we put non-important ESDs on cold storage then we're back to the current problem as well, storage costs. Cold storage is just as expensive, if not even more expensive, especially the initial cost. (Anyone want to donate an Ultrium tape drive allowing for a minimum of 1TB (uncompressed) tapes? No, FYI these costs some 3000+ USD). Also, our goal is to have everything online and accessible for our members.

Well let's say keep the main lab and have others offline.

Blu-rays are a good cold storage solution, Facebook trusts it
http://datacenterfrontier.com/inside-fa ... ta-center/
High quality 50GB Verbatim discs cost $48 per TB
http://www.ebay.co.uk/itm/20-1-FREE-Ver ... 1e61dad4b1

Not the best solution but I suggest it so ESDs aren't lost in case ISOs are chosen.

[mrpijey]

So what makes the main lab more important than others, from a preservation collecting perspective?

And no, blurays are not a good cold storage unless it's stored under optimal circumstances, such as in temperature controlled environments. I've already have a couple of BDRs break for me with reading errors etc...

And 48 bucks for 21x50GB? That's 1.05TB total storage, you do realise that a 1TB harddrive costs about the same, which is by far safer storage than plastic discs., and by adding around 30 bucks more I get 3TB.... so no, plastic discs are not very cost effective, especially not if you consider you would need to make double copies to make sure that one of them doesn't die of bit rot...

[Hackerpcs]

Nothing, I'm not saying to delete other labs but have them offline and available by request if costs are the problem.

I wouldn't trust HDDs, spinning magnetic surfaces, for long term archiving, in case you leave your discs in the sun (???) they will not deteriorate if they're in a dust free (cake boxes inside a carton box e.g.) normal temp environment, providing both burner and discs are high quality. I've burnt many (over 100) Verbatim BDs with LG GGW-H20L since 2008, not one broke and function perfectly, on the other hand I have 2 WD HDDs fail and 2 WD HDDs replaced, hopefully with their data saved.
5-7 years for non-tape media is a good period (not that I've seen any disc break after 7 years) because after that you can migrate with much lower cost to then bigger media.

[mrpijey]

But moving the other labs won't do much in terms of storage. I am talking about the ESDs here, not ISOs. The ISOs deduplicate well as they are, and other labs won't do much difference.

Well my experience is the opposite, over 10 years of harddrive usage I've lost only two drives (one was due to my own fault), and the other was an IBM drive... whereas I've lost several discs, even when they had been stored in a dark place and not touched for years. A drive that occasionally is spun up and used should last for many years.

And 5-7 years is a really short time, I've got 24/7 active harddrives older than that still working and 100% error free. All active drives have offline backups so if they fail I always have data recovery, but for these ESDs I rather chuck them all onto a really big drive than spread them around in a box full of plastic discs. Also, the more complex the discs become the more sensitive they are as well, a scratch on a CDR barely makes a dent, but a speck of dust on a BDR makes it unreadable as it covers a much larger data surface.

Remember also, by doing the ISO conversion I can use dedup and basically fit it all on a single drive, and then I can also afford a secondary backup of it, making sure that it's never lost. And all ESDs are already packed up with recovery redundancy to avoid bit rot etc.

[q3hardcore]

Why not decrypted ESDs?

[Wheatley]

Why not decrypted ESDs?

Because those would not be original anymore.

[hounsell]

It's not like much of the ESD is actually encrypted anyway. Only a few key parts, like headers, etc. A fully encrypted file would take hours just to decrypt on atom tablets and the like.

I suspect the reason it's not very conducive to deduplication is because ESDs also use a different (better) compression algorithm than the WIMs that ship in ISOs.

[jagotu]

Doesn't the esd stay as install.esd in the converted ISO anyway?