Suggestion/Opinion regarding SFV files

New news and release discussion.
Post Reply
SistemaRayoXP
User avatar
Posts: 371
Joined: Wed Sep 13, 2017 1:26 am
Location: Tlajomulco de Zuñiga, Jalisco, Mexico.
Contact:

Suggestion/Opinion regarding SFV files

Post by SistemaRayoXP »

In a recent poll, BA Admins asked whether to keep Windows 10 betas since they were holding too much space. The final decision was to reduce the space by taking a base build (en-us), compare it to its localized counterparts (en-gb) and save the differences with SmartFileVersion files (SFV), so that we can rebuild the localized ISOs using SmartFileVersion. To optimize the reach of this measure, this was done to all ISOs possible, not only Windows 10.

The last week or so, I needed to download Windows Vista SP2 es-es, and I found that there was no es-es, but rather the en-us + an SFV which contained the SFV for building the es-es ISO (plus many other localized ISOs) that was more or less 10 GB in size. The fact was that I spent over 10 GB of my daily quota in downloading a lot of SFVs, of which I only needed one, which was 233 MB in size or so.

It's not a big deal, as I download ocassionally some GBs, not even getting any close to download 20 GB+. But not only I'm spending more than needed quota, I'm spending FTP more than needed bandwidth, and bandwidth costs. So the solution, in my opinion, would be to offer each file individually. I foresee the problem being that the compression might not be as effective (taking in mind all SVFs are saved as a big RAR), but with the extra space, it might not be so much of a problem (I guess?). Of course other than the effort needed by mrpijey to do this task.

Thanks in advance for reading.

yourepicfailure
User avatar
Donator
Posts: 1317
Joined: Mon Jul 23, 2012 9:40 pm
Location: Lufthansa DC-10

Re: Suggestion/Opinion regarding SFV files

Post by yourepicfailure »

I do agree that downloading the entire SFV does take up a bit of size+bandwidth, and is unfortunate for those of us with limited bandwidth.
For instance, I myself wanted to try the DE-DE of a Win10 build (don't remember right now what build) but the sfv took on ~4gb IIRC.
It is a bit of a bittersweet thing. They may be in rars to help mitigate some corruption as well as simplify organization and databasing. Having them in another folder on FTP rather than a single .rar would be a suggestion. But mrpijey would have to swoop in for why or why not it would pe possible. Compressing them does save a couple hundred of megs.
Like I said, I would like to read mrpijey's explanation for further clarity as to why they were compressed as a single archive.
"C makes it easy to shoot yourself in the foot; C++ makes it harder, but when you do it blows your whole leg off"
Image
You will never tear me from the grasp of the Pentium M!

mrpijey
User avatar
Administrator
Posts: 9188
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: Suggestion/Opinion regarding SFV files

Post by mrpijey »

I understand the issue, and I factored in these problems when I created the svf combo files. I combined them simply to make it easy, and to not create any confusion of which one was the source file for the patches (since it's always included with the patches as media_iso.rar). It would also cut down on the amount of releases taken up by Windows 10 etc. But I do see the problem since you would need to grab it all to get a single language. Sure I could separate it all into separate files, but it would create a massive amount of additional releases, and members would also be confused which ones was the source file. It would also add additional tags to the patch files which is problematic enough with the limited path length the FTP software supports... And since the svf files are patches to a source file and not a separate release it would be hard to separate them from its source release too, so I am not sure how to have it both ways. I could possibly put all the patches in separate rar files as a subfolder, but that would also increase the path length far beyond its limitations and break the entire procedure of how folders are read and managed by the scripts... so it's not easy to solve.
Image
Official guidelines: Contribution Guidelines
Channels: Discord :: Twitter :: YouTube

yourepicfailure
User avatar
Donator
Posts: 1317
Joined: Mon Jul 23, 2012 9:40 pm
Location: Lufthansa DC-10

Re: Suggestion/Opinion regarding SFV files

Post by yourepicfailure »

Maybe, taking into consideration limited length, compressing the svfs into .rar in a subfolder?
E.G. /[whatever build]/svf/deDE-HomeSVF.rar

This would cut back on path lengths as well as bandwidth.

The concern I see is the work required to individually compress and verify each rar. In addition to databasing.
However, an idea for the databasing is possibly centealize the builds. For instance, I search 9834. I get one link to the build tag, then on the page for it state what's available like platforms, SKUs, and languages rather than many results for eachbuild. For instance a search for 9834 yields 6 results for the Win10 build. The possibility would make that into 1 result for those 6, containing everything about that build. However, this may prove inplausible due to the current database structure.
"C makes it easy to shoot yourself in the foot; C++ makes it harder, but when you do it blows your whole leg off"
Image
You will never tear me from the grasp of the Pentium M!

mrpijey
User avatar
Administrator
Posts: 9188
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: Suggestion/Opinion regarding SFV files

Post by mrpijey »

Regardless what solution we try it would need a lot of rewrite of the code I have that processes releases, and it would also require a massive resync of two backups, something I am not willing to do at this time. I've been repacking and resyncing for over 6 months now just to get everything right and make sure that we both save space and bandwidth, and I don't want to redo half of it again.

The only thing I can think of is to repack the patches individually, but the problem then is that we will then unassociate the patches with its source file, risking we lose the entire build if something goes wrong, this is why I bundled them together. Any other solution to keep them together would require a massive redo of the FTP layout too, something I am not keen on doing at the moment. If we had a web download system then it could be easily done as the layout wouldn't matter, but we don't have any of that yet. One other idea is to simply have an another subfolder (media_svf) but then we will extend the path further which will cause issues with a lot of FTP clients. And I have no way of shortening the path unless we shorten the root paths, something I've also considered.

I am afraid that for the time being we have to keep it the way we have it today, it's better than nothing, and far better than before (which took up tons of more space and was much harder to manage). I will continue to experiment with various solutions to make it easier for the users to grab the exact ISO or patch they want, but it will require a lot of work either way, something that will take a long time as we're talking about thousands of files, with the added backups and syncs.

Thanks for bringing this to my attention and I will continue to find a solution. If any ideas come up please let me know.
Image
Official guidelines: Contribution Guidelines
Channels: Discord :: Twitter :: YouTube

Post Reply