BetaArchive Logo
Navigation Home Database Screenshots Gallery Image Uploader Server Info FTP Servers Wiki Forum RSS Feed Rules Please Donate
UP: 56d, 15h, 21m | CPU: 57% | MEM: 5965MB of 12287MB used
{The community for beta collectors}

Post new topic Reply to topic  [ 16 posts ] 
Author Message
 PostPost subject: Xbox OS [Original/360/XBone]        Posted: Sun Sep 30, 2018 1:28 am 
Reply with quote
Donator
Offline

Joined
Sat Sep 09, 2006 6:43 am

Posts
773

Favourite OS
Win10/Debian Linux
Let's talk Xbox!

Disclaimer: I will not provide links to sources, as per forum rules. You can do your own googling.

The Original Xbox Console's Operating System, herein called xboxkrnl, was based on Windows 2000 code. It was designed around a heavily customized Windows Embedded line of Operating Systems.
Internally, the OS consists of the bare-minimum boot code to transfer from x86 POST up through switching to protected mode and 32-bit kernel mode. The entire kernel exists within the first 1MB of system RAM once it's fully booted.
Note: There are security systems in place that have been both defeated and bypassed due to failure to read Intel documentation on Microsoft's part. Those are not things that I will talk about in this post, as they are secondary to the main idea behind this thread.

The underlying Kernel was designed specifically for the Xbox. There has been lots of speculation as to whether or not the OS was completely written from scratch, or if it was simply a modified form of the NT kernel. During my own research, I compared the source tree directories and files stored within to the Windows Research Kernel, and found that, for the most part, NT and XboxNT are very close cousins. A large amount of core services were ripped from the NT source and given a home within the Xbox OS. That isn't to say that those services weren't modified for a low-memory environment, though.

The OS was designed from roughly 1999 through 2001, with Alpha SDK kits going out early on in 2000. These kits were basic PC's with a specific target hardware feature-set. The earliest of the SDK's, which seem to have never appeared up for auction, were AMD-based systems. Due to Intel offering a lower price (a deal) for processors, subsequent SDK machines came with Pentium III's. nVidia provided the graphics chipset for all the machines. Even after AMD was dropped for processors, their AMD-760 chipset was kept around for the final hardware design. The Alpha II SDK, however, utilized an Intel 440BX chipset, with a standard Intel Desktop motherboard.

These Alpha SDK's utilized a Windows98 DOS7.1-based loader to get up and running. The earliest form of the Xbox OS was chain-loaded via DOS. From 16-bit land to 32-bit land, Xbox was just any other NT. Included on the recovery disk for the Alpha II (Build 3146), are DOS files, as well as an xboxldr, XboxKRNL, and various support files for the Xbox dashboard, as well as a debugger.

One of the biggest differences between NT and XboxNT is the lack of a Windows directory, separate driver files, and/or any external resources for the running kernel. Everything that is necessary to run the kernel, drive devices, etc., is stored within a single kernel executable. This was by design, as their target ROM size was 256kb. They did use compression (Specifically LZX) to make use of the lack of ROM space more efficient, which allowed the compiled kernel to exceed the small constraints of 256kb. As part of the security system, they moved the real 16-bit bootloader into the nVidia MCPX chipset, put a dummy, highly-compressible fake bootloader in the ROM, which would get overwritten at boot time in memory, and added a 32-bit bootloader to the top of the ROM to bootstrap the 32-bit kernel. This allowed for a very quick startup, as nothing was needed to be stored on optical or magnetic media for the system to boot into XboxNT.

*XboxNT is what I call the OS. I haven't seen it known as such in official documentation, but it IS a possibility that it was known as such within Microsoft

The kernel revolves around an embedded x86 target with DirectX capabilities. It may have been the most successful NT product/project prior to Windows XP. Under the hood, DirectX 8.0 was targeted, fully utilizing all the hardware functionality provided within the Geforce3 embedded chipset. nVidia also provided an early preview to the nForce Soundforce audio processing unit within the chipset. (Of which, is only now being researched heavily due to emulation efforts; It has a dedicated Texas Instruments DSP, with lots of intricate features that not even nVidia or Microsoft ended up utilizing).

The original name for the project was the "DirectXBox". The entire reason for its existence was to utilize DirectX for a dedicated gaming machine

Inside the kernel, several drivers and services are cut down so that the system does not need external resources to run - no Registry, for instance. This is what makes it one of the most efficient NT-based OS's ever. It only does the bare minimum to give a sane execution environment. Several services that would normally be provided by Windows, like directX runtimes, aren't present. The compiled games contain the majority of the code for DLL's, which allowed an older Xbox to continue doing new tricks. This design was not replicated for the later consoles, however. Perhaps it was due to a worry of security - allowing a user to modify a game's executable with their own libraries would provide an easy path to exploiting the system.
This also allowed newer games to utilize the hardware on a lower-level than the normal NT ecosystem would allow.

Interestingly, however, video drivers are provided in the kernel, stuck at whatever version nVidia was up to at the point in time that the kernel was compiled. As far as I can tell, nVidia was already done with providing support for the Geforce3-based chipset by the time the Xbox launched - late 2001. Even though they continued to release Detonator drivers for desktop-versions of the chipset, the stable-state that the Xbox provided shows just how little changed within the DirectX 8 era.

The Xbox only has 64MB of RAM to use. This includes just 16MB sectioned off for graphics memory, 1MB for the kernel, and the remaining 47MB for game code to execute in. This does not take into account the Swap space on the hard drive, which consisted of 3 partitions of roughly 700MB. Several games were written to fit within that 47MB space, while others -- typically the ones that really shined above the rest -- used the swap partitions as a cache for storing large files from the game discs temporarily.

The original Xbox could not have benefited much from DirectX 9, as the hardware feature level was stuck in version 8.0. That didn't stop developers from trying to exploit low-level hardware features to utilize DirectX 9-era/style shaders. Most of the later titles pushed the hardware to the limit. Halo 2 was one of these titles, that, even when released, didn't play well on the hardware due to the constraints of 64MB of RAM. Developers of Halo 2 have said that the game really only ever ran perfectly on DevKit Xbox's, which had 128MB of RAM, allowing for an additional 64MB of space for game code. There was no additional allowance for texture memory, however.

As the Xbox aged, the final Xbox Development Kits came with newer libraries, of which could be linked with a game, to provide newer experiences on the old hardware. This allowed the old hardware and underlying Kernel to become host to new code.

With the Xbox 360, came a complete departure of sorts. NT on PowerPC had been done before, during the NT4 era. Apparently that development had not ended internally, as tooling up for the Next Xbox was done quickly, and Apple PowerMac G5's became the basis for Xbox development, now powered by DirectX 9.0 features, using an AMD R500-based chipset. The Xbox360 OS is something that I haven't spent much time researching, but it does appear to be based on NT5 from a cursory glance. Given the timeline of OS development, it is likely based on NT5.2 (2003) code, and, like the original Xbox Kernel, is designed to run within a small footprint - 16MB this time around. However, given the security systems in place, a kernel backup is always present, as is the entire dashboard, so that size quickly diminishes to 12MB of usable flash storage. Later Xbox 360 consoles came with 256MB flash ROMs to store the entire dashboard in ROM, and provide storage for an entire Kernel + Dashboard backup in flash.

The Durango was a complete departure yet again. This time, Microsoft chose to base the Xbox kernel off of Windows 8's kernel. For the earliest Durango SDK, they simply took an interim version of Windows 8.0/8.1, stripped out the Win32 Subsystem, and placed in the Xbox Subsystem, based on DirectX11. The Xbox Subsystem at this point in time, was a culmination of the Xbox 360 OS Services, DirectX, and drivers for all the hardware in the system. The underlying OS is anything but stripped down - at least the devkits appear to be completely off-the-shelf components. Granted, they added security layers upon security layers in the actual final product. The original Durango vision was a PC-based machine with several layers of virtual machines to run the kernel and system services off of.

During Windows 8.1 to Windows 10, the Windows RunTime sub-system was under rework and re-development. Alongside WindowsRT, the Xbox SubSystem was developed as another flavor of RT. The Xbox One runs a Windows 8.1 through 10 base kernel, followed by several encrypted hyper-v containers for every program in user-space. This allows for the hardware to change over time, yet provides a stable target for developers of games and apps to develop for. As of writing, Windows 10 RS5 is set to launch any day now, and as per buildfeed.net, the Xbox One is also receiving RS5 any day now.

Microsoft has shifted Xbox OS development to a SubSystem of the NT OS. It's been quite a ride, from having to develop for a hardware target, to now having to develop for a software target. The underlying hardware means less today than it did 17 years ago when the project got going. Virtualization has helped push the boundaries of the entire gaming ecosystem.

_________________
Need disks scanned in the USA? I have a Kryoflux, and am willing to help get your disks archived! I also offer xbox and xbox 360 repair and modding services. PM me for details!


Top  Profile  WWW
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Tue Oct 02, 2018 12:07 pm 
Reply with quote
Donator
User avatar
Offline

Joined
Mon Nov 20, 2017 6:52 pm

Posts
71

Favourite OS
Longhorn 4074
The Xbox wasn't the first game console to be capable of running a Windows operating system and have DirectX features, the SEGA Dreamcast allowed developers to use Windows CE as the environment and utilize its DirectX APIs. It wasn't that well received, but specifically PC developers used it to make porting PC games easier. It's very interesting to see how the SEGA Dreamcast was Microsoft's testbed for their eventual Xbox console.

_________________
MCSA: 70-410, 70-740, Currently studying for 70-741
MTA: 98-349, 98-365


Top  Profile
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Fri Oct 05, 2018 12:04 am 
Reply with quote
Donator
Offline

Joined
Sat Sep 09, 2006 6:43 am

Posts
773

Favourite OS
Win10/Debian Linux
DanielOosterhuis wrote:
The Xbox wasn't the first game console to be capable of running a Windows operating system and have DirectX features, the SEGA Dreamcast allowed developers to use Windows CE as the environment and utilize its DirectX APIs. It wasn't that well received, but specifically PC developers used it to make porting PC games easier. It's very interesting to see how the SEGA Dreamcast was Microsoft's testbed for their eventual Xbox console.


The Dreamcast ran on a heavily customized version of WindowsCE, which was a cut-down version of Win9x. I'm not entirely sure on the specifics for the underlying OS, but I suspect it was 9x based and not NT. As far as I recall,it didn't utilize DirectX, but OpenGL, via the PowerVR chipset. I don't believe that the underlying WindowsCE kernel exposed any DirectX API's, but I could be wrong. It wasn't well received in the US, but thrived for quite a while in the Japanese market. It also powered an entire generation of Sega Arcade machines, until the Chihiro came along (which was, in fact, a debug-kit Xbox connected to a Jamma interface and GDROM drive).

Edit: did some more research. The Dreamcast did support DirectX, but it also supported OpenGL for platform-specific releases. The disadvantage was a slight performance hit, but it was preferred since it made porting games a lot easier and cheaper.
-----

nVidia utilized the Xbox as a test platform for their upcoming nForce chipset. Microsoft utilized it as a means of utilizing its software IP in hardware form. It was a good deal for both companies. Sega didn't get the market backing that Microsoft could have provided, especially in the US market -- and it's likely because Microsoft had already begun on their own game console, using newer embedded OS solutions that wouldn't become part of the WindowsCE/Windows Mobile toolchain until 2003, with Windows Mobile 2003, which was the first NT-based Windows Mobile release.

At the time of the DirectXbox's creation, Windows 98 had just shipped with DirectX 6, and development had begun on the next version which would help push DirectX into a targeted platform for GPU and audio chipset manufacturers. DirectX 7 shipped with Windows Me, with DirectX 8 following not too long after. The final hardware utilized DirectX 8.0, and not the later 8.1, which would be supported in later cards from nVidia, like the GeForce 4. The silicon in the Xbox is an embedded cousin of the original GeForce3 chipset (not the Ti 200 or Ti 500 variants).

_________________
Need disks scanned in the USA? I have a Kryoflux, and am willing to help get your disks archived! I also offer xbox and xbox 360 repair and modding services. PM me for details!


Top  Profile  WWW
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Fri Nov 16, 2018 1:36 pm 
Reply with quote
Donator
Offline

Joined
Sat Sep 09, 2006 6:43 am

Posts
773

Favourite OS
Win10/Debian Linux
As I delve further into researching the underlying libraries that the Xbox OS was built on, I've come to find that the shipped sdk includes are from NT5.0. It's interesting to see some form of confirmation that XboxNT was a form of the Windows 2000 kernel - which had limited Embedded application, short of "Windows-Powered", which was itself just an addon for the Workstation SKU, and not a Platform like NT4 Embedded was.
The DreamCast differed in this area, as it utilized a compacted version of the WindowsCE kernel, of which was completely separate from 9x or NT. It was platform-targeted, with very little libraries stored in ROM, and each game would come with an executable with the kernel linked into itself. The DreamCast itself didn't run WindowsCE, but a limited firmware that offered pre-boot capabilities, and full hardware control. WindowsCE provided a HAL along with DirectX API's to control the underlying hardware. It required that the underlying firmware provide bootstrapping, and basic hardware IO functionality. It was also capable of running its own software, running in OpenGL on the PowerVR chipset. The DreamCast was a testbed of sorts for Microsoft - while not footing the bill for hardware design and marketing.

The XboxOS was all that the Xbox came with, in regards to ROM. The Xbox does very little in hardware to boot the system, which is comprised of only the 1st bootloader and 2nd bootloader, hidden in the MCPX chipset. The 2nd bootloader only executes after ROM contents are verified via a cryptographic key stored in the MCPX. There's safeguards in place both in hardware and software to prevent reading of the MCPX ROM, which is one of the issues with emulation of the Xbox. As of writing, there is no way to pull the 1st and 2nd REAL code from the MCPX, as one of the safeguards is to mask the real MCPX ROM by overwriting it with a fake 2nd bootloader stored in the ROM - And, as it stands, the only way to pull this data would be to write our own crypto-signed 1st bootloader that is signed with original Microsoft keys, which no one has except for Microsoft themselves.

There are two projects with two different takes on emulating the Xbox.
Cxbx-Reloaded, which 'simply' patches game executables with corresponding Win32/DirectX API's. Under the cover, however, Cxbx-Reloaded includes its own implementation of the Xbox Kernel. They currently have lists of patches for supported and unsupported games, which the software scans through and applies on-the-fly. Many games work, some fully playable. Many games don't work, some crashing the software entirely. System speed no longer affects emulation, as the software tries to run games within the original time spec as the original xbox.

XQemu is a complete hardware emulator. It is designed to utilize all the original MCPX ROM + FLASH ROM software from the Xbox, and provide graphics emulation based on the original NV2A chipset in the Xbox. So far, it provides incomplete 1:1 emulation of the Xbox hardware. Currently missing is sound support. This is due to the fact that the emulation is based entirely on clean-room reverse engineering, and not utilizing patching of executables like CXBX utilizes. Upon starting XQEmu, you need to have the MCPX ROM, and an XBOXKRNL of some sort - one of these components is hard to come by, and is currently not available via any software means. The XBOXKRNL can be sourced via Cromwell, a linux-based drop-in replacement. Unfortunately, Cromwell does not provide DirectX API's, so only homebrew apps written against the Cromwell API's can execute.
The issue here is legality. Microsoft made it extra hard for reverse engineers to implement the Xbox in software. From this standpoint alone, Cxbx-reloaded is legally in a good spot. It doesn't use any Microsoft code. It is clean of all possible legal issues regarding using ROM from an actual Xbox console.

At this point in time, Microsoft has announced the next Xbox, code-named Project Scarlet. They're currently claiming backward compatibility. Whether or not that just means Xbox One backward compatibility, 360 compatibility, or Xbox OG compatibility, no one can be certain. There's currently speculation that it means 100% compatibility. This has yet to be confirmed, but it could mean that Microsoft is actively developing a 1:1 Xbox compatibility layer for the modern Windows 10 Embedded platform that the Xbox One runs off of. Should that be the case, work on reverse-engineering the Xbox Next's software should be considered, as it should be possible to rip the packages from the Embedded platform and migrate them to a running Windows 10, thus providing a semi-legal, fully Microsoft-implemented emulator.

The next question is whether or not Microsoft will provide this support in the Windows 10 Xbox app itself. Whether or not Project Scarlet will include porting this functionality back into the Windows 10 ecosystem is a big question - they may actually provide emulation of Xbox and Xbox 360 titles on the PC. It's a slim chance, but one that may be part of Microsoft's plans for the continued development of Xbox products.

_________________
Need disks scanned in the USA? I have a Kryoflux, and am willing to help get your disks archived! I also offer xbox and xbox 360 repair and modding services. PM me for details!


Top  Profile  WWW
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Tue Dec 18, 2018 3:08 pm 
Reply with quote
Donator
Offline

Joined
Sat Sep 09, 2006 6:43 am

Posts
773

Favourite OS
Win10/Debian Linux
The more I look into the "XOS" kernel, the more I learn C syntax, and see some interesting bits.
First and foremost, internally, the Xbox OS is named XOS. It is a lightweight version of the NT5 kernel, which also utilizes minimalized windows NT 4 drivers for hardware.
The bootloader has code to boot the kernel from DOS(!!)

The Alpha I and Alpha II machines actually did boot the kernel from DOS, and included an xboxrom.bin on the update CD's. I believe it may be possible to boot the kernel from DOS - however, the kernel expects certain hardware at certain PCI slots, or it will panic and crash. I really should have kept my Pentium III system around for testing :(

The kernel assumes an Intel 440BX chipset is in use -- even within nVidia-code; The MCPX chipset was an intel 440BX-compatible chipset, despite the assumption that it had been an AMD-specific design. There are bits and pieces of leftover AMD-targeted code, but it's either unused or so generic that any x86 chipset will run it.

Ultimately, XOS is a very lightweight NT-like OS, but its own variant that only uses some parts of the Native NT code to bootstrap the system. Upon loading the kernel, the system is immediately running the Xbox Subsystem, and does not utilize any Win32 API's. Almost all API's are provided by program executables - XAPI is linked against the kernel, but is exposed in various versions in programs themselves.

_________________
Need disks scanned in the USA? I have a Kryoflux, and am willing to help get your disks archived! I also offer xbox and xbox 360 repair and modding services. PM me for details!


Top  Profile  WWW
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Tue Dec 25, 2018 1:57 pm 
Reply with quote
FTP Access
Offline

Joined
Tue Feb 13, 2018 2:27 am

Posts
26
This is quite the read. It's pretty interesting how they manage to fit everything in such a small space digitally. I feel like its becoming more of a lost art with the massive storage space we have now days.

I do have a question though. Is it possible to modify the Xbox's OS to be able to run on a regular PC, or is that just technically impossible?


Top  Profile
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Tue Dec 25, 2018 7:59 pm 
Reply with quote
Donator
Offline

Joined
Sat Sep 09, 2006 6:43 am

Posts
773

Favourite OS
Win10/Debian Linux
There were efforts years ago to implement the xbox api into Windows. I don't think they ever got anything working, though.
Technically, it might be possible, albeit very, very hard to implement. The XAPI is undocumented. The few efforts to document it are typically found in Cxbx-reloaded, which is the only somewhat fully-working emulator for the Xbox on Windows, aside from whatever it is that Microsoft has planned with Project Scarlet.

The original xbox hardware, despite it being based around PC hardware to a degree, is customized enough to be a completely different animal. It's an embedded x86-compatible platform, and expects certain hardware at certain places - pci hardware, video hardware, memory, etc. There's lots of layers on top of the OS intended to verify that the code is secured, too. It's not as simple as dropping the kernel in to an embedded Windows and turning the machine on. Lots of little dependencies that things like XQemu emulate are necessary to get things going.

That isn't to say that it's impossible, but the work needed to make it function with what would be thought to be minimal effort is immensely difficult.

The xbox has its own variant of NTLDR in 3 parts. It has a 16bit preloader, a 32bit bootloader, and a 32bit verify stage bootloader, which checks the keys of the system against a key stored in the hardware (MCPX). If one of these stages fail, it doesn't load. Getting the appropriate keys is the hard part. Having one key isn't enough - there's a chain of trust between all the bootloading components.
The preloader has a key, the bootloader has another key, and the "2bl"- the 32-bit verify stage bootloader, has - you guessed it - another key.

The device drivers are a whole other level of complications. The original kernel builds off of a customized nVidia driver. It has special tie-ins to the underlying kernel. From my research, it looks like Microsoft took their basic video driver from the DDK sample, and customized it to utilize nVidia's customized driver. It's a mix between Windows NT4 and NT5 drivers - likely due to the lack of a registry to store settings in. NT4's driver model is much simpler, but also a lot more temperamental to hardware changes. Everything is hardcoded.

The lack of a registry means that everything is tightly knit around the xbox hardware. Taking the hardware out of the equation and building upon this NT-like system might be possible - but again, the amount of work necessary would likely be negated by other attempts at implementing the XAPI on top of Windows - which, is very likely why Cxbx exists in the first place.

_________________
Need disks scanned in the USA? I have a Kryoflux, and am willing to help get your disks archived! I also offer xbox and xbox 360 repair and modding services. PM me for details!


Top  Profile  WWW
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Thu Dec 27, 2018 3:24 pm 
Reply with quote
FTP Access
Offline

Joined
Tue Feb 13, 2018 2:27 am

Posts
26
Thanks for the fairly detailed answer. I figured hardware could be an issue, I just didn't know to what extent. It seems Microsoft really took all of the steps here to ensure the system would function well with the limited resources it had.

As far as emulators go, do you know how Microsoft is getting original Xbox games to play on the One?


Top  Profile
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Thu Dec 27, 2018 8:34 pm 
Reply with quote
Donator
Offline

Joined
Sat Sep 09, 2006 6:43 am

Posts
773

Favourite OS
Win10/Debian Linux
Microsoft has provided no details on their project. We don't know if they're just using the old 360 emulator, or if it's a native emulator using hyper-v. I'd guess that they're employing a highly customized hyper-v implementation for optimal results, which is why it's taking so long to get it going.
The 360 emulator for OG xbox was sub-optimal, as it required profiles, on a per-title basis. In some cases, xbox live would pull a different executable for the game, in xex format.
The xbox one's architecture utilizes hyper-v containers for everything it runs. The dashboard runs in one container, games, apps, etc in others. It's likely that they will employ this in their OG Xbox offering, in a customized hyper-v variant.

_________________
Need disks scanned in the USA? I have a Kryoflux, and am willing to help get your disks archived! I also offer xbox and xbox 360 repair and modding services. PM me for details!


Top  Profile  WWW
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Tue Jan 08, 2019 10:32 pm 
Reply with quote
Donator
Offline

Joined
Sat Sep 09, 2006 6:43 am

Posts
773

Favourite OS
Win10/Debian Linux
Continuing on other fronts of research, I've begun decoding the XE format, which the Alpha II used, as well as a title that was found on the ftp. It's not quite a Windows PE32-format format. It doesn't follow the spec. I've managed to get a bit of 010 template magic to figure out the file format. It appears that several bits are either hard-coded in the file header, which does not apply to the PE format. It also does not have the old DOS MZ header, and starts in the PE header itself. Then there's a few sections of the PE header that are missing entirely. I started from the shipped 010 template for EXE files, which may not have been the most ideal starting point, but it does make it easier to remove things that are outright missing from the XE specification.

At this point, I've isolated the sections, and verified that the data stored in those structures is valid. VirtualAddresses, for instance, point to the right places in the file for the section data.

On the section data - the values for the NumberOfSections are not located in the main header, like the PE format - they're located in 3 repeating regions of the XE - at 0x44, 0x46, and 0x48. Using two samples of this file format has helped in seeing differences between the alpha XE format and the (presumably) debug XE format. Then there's a chunk of data between the header and the sections start, which I'm unsure of. It contains the appropriate NumberOfSections data, and a bunch more data that I have yet to determine.

Ideally, knowing the format would allow someone (not me) to write a converter to the standard XBE format, and potentially make these samples run under in an unsigned xbe loader (CXBX-RELOADED, XQEMU). Or at the very least, analysis would be possible in a tool designed for XBE's. Documenting the file format may be beneficial for preserving historic builds of games that we have here at BA, and for future leaks (should there be any).

_________________
Need disks scanned in the USA? I have a Kryoflux, and am willing to help get your disks archived! I also offer xbox and xbox 360 repair and modding services. PM me for details!


Top  Profile  WWW
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Tue Jan 08, 2019 11:29 pm 
Reply with quote
Donator
User avatar
Offline

Joined
Mon Nov 20, 2017 6:52 pm

Posts
71

Favourite OS
Longhorn 4074
Have you seen the recent XenonWiki? It's a new Wiki dedicated to documenting the 360's beta operating systems and hardware, and recently they have released a guide how to build your very own PowerMac G5 "FrankenXeon", along with a hard drive image of XenonOS for the Alpha G5 kit. Apparently it is capable of emulating Original Xbox titles too, but it's far from perfect and likely won't work well with a majority of OG titles, if even at all.

_________________
MCSA: 70-410, 70-740, Currently studying for 70-741
MTA: 98-349, 98-365


Top  Profile
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Mon Mar 18, 2019 7:31 pm 
Reply with quote
Donator
Offline

Joined
Sat Sep 09, 2006 6:43 am

Posts
773

Favourite OS
Win10/Debian Linux
Apparently I cannot edit my own first post to correct some mis-information, so I'm adding it here.

The Xbox executable format actually contains the necessary Drivers for the nVidia chipset. The XboxNT kernel/HAL only exposes the GPU hardware to whatever library an XBE contains. In turn, newer games were able to include newer drivers (within DirectX libraries), and offer more features. The XboxNT kernel does not have much in the way of video software, and only has a handful of support libraries linked in to the kernel itself. These libraries include VESA graphics extensions (things that would normally be part of a video card bios), and the base XGRAPHICS library. Once games are executed, the drivers included in the XBE are loaded by the kernel, and full control of the hardware is transferred to the XBE program. The kernel only allows a single process to run at any given time. All games run in kernel mode, which is why the main subsystem is XBOX and not NT or WIN32. There are bits of WIN32 in the kernel for executing the XBE loader and bits within the stripped down kernel itself.

The kernel source code directly is structured the same way as the NT source code directory, however, xbox-centric bits tend to have an "x" appended to the re-written parts of the kernel. For instance, for the \video\ directory, where we'd normally see \video\port, we have \video\portx. This is a specially written version of the video port, specifically setup so that the kernel exports exist not in a separate VIDEOPRT.SYS file, but within the kernel itself.

Several other XboxNT-specific bits are included, such as the bootloader parts, in \bootx, the filesystems drivers in \fatx and \gdfx, the HAL within \halx, and many more. The \video\portx code is obviously built from the NT4 DDK, as dates and comments are unchanged from the samples shipped in the DDK. The video drivers that are seemingly included for support, are Windows NT 4 drivers. The HAL bits needed from the nVidia code are pulled into the kernel from an NT4 driver!

The version of the included build tools are specialized for the XBOX subsystem. They still retain standard NT and WIN32 personality support, should the code need to be compiled for those platforms. Kernel32 exists in some fashion, deep within the kernel itself. The Xbox subsystem utilizes DirectX on top of WIN32 API's. It's an extrapolation of the DirectX 8.0 API set, streamlined to be as efficient as possible within the small constraints of the target hardware. Remember when I mentioned that libraries and drivers are included in XBE's? DirectX's APIs can be extended within the XBE's provided libraries. This is one of the smart decisions Microsoft made - the ability for games to target a slightly newer DirectX API (8.1), without needing to upgrade the base kernel. Even though the Xbox comes with a FLASH chip for the BIOS, the ability to re-flash it was removed (albeit you can re-enable it by soldering a few points on the motherboard). Microsoft never intended for the base kernel to receive any updates -- the libraries inside the XBE's, on the other hand - were.

Microsoft released several updated XDK's throughout the lifetime of the Xbox, and in doing so, was able to add new features to the aging architecture. Bungie, developers of Halo and Halo 2 for the original Xbox, did note during the E3 demonstration of Halo 2, that they were pushing the hardware as far as it could go. This actually meant that they had utilized the latest XDK, and all of its features available to utilize all the Xbox had to offer.

It may be worth looking at XBE's in the future, to determine exactly what the very last release of the XDK's libraries were. As far as I know, the last game was either Madden or some hockey game, released in 2008 or 2009. The last known XDK release was in late 2003.

_________________
Need disks scanned in the USA? I have a Kryoflux, and am willing to help get your disks archived! I also offer xbox and xbox 360 repair and modding services. PM me for details!


Top  Profile  WWW
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Thu Mar 21, 2019 2:57 am 
Reply with quote
Donator
Offline

Joined
Sat Sep 09, 2006 6:43 am

Posts
773

Favourite OS
Win10/Debian Linux
XboxNT was really not based on Windows 2000 RTM code.

The directory tree doesn't match extrapolated directory trees from leftover strings in debug executables for Windows 2000 RTM.
The video subsystem in 2000 RTM was moved around from ntos\video to another part of the kernel entirely.
The XboxNT directory tree appears closer to NT4's directory tree. This also goes hand in hand with the inclusion of NT4-compatible video driver data.

Seeing as development on the Xbox began in late 1998, it is likely that they simply pulled the current unstable code (NT5), before a refactoring occurred, likely sometime later when WDM became the next big thing. XboxNT's kernel doesn't utilize any form of WDM tech, as all drivers are loaded from XBE libraries, and aren't included with the kernel - they're stripped of anything not absolutely necessary for functionality.

It is my conclusion, at this moment in time, that the Xbox variant of Windows NT is neither 2000 nor NT4, but beta NT5 code. The common myth since the release of the Xbox is proven untrue.

Now, what can be learned from this? Not much. The source for the XBE libraries never leaked, as far as I know. That's where the magic really happens. More so than the kernel source code. Microsoft managed to create a 'minwin'-esq kernel for the Xbox to boot off of - long before the minwin project took place. They built the best possible base OS for a gaming console - and ended up extending it with the Xbox 360. It's likely that they built the earliest xbox 360 code off of the original Xbox's code, as code paths still existed in the makefiles for PPC and IA64. That alone should have been a clue that the Xbox wasn't NT 5.0.2195 based.

Going backwards in time, on this very forum, there was talk about NTOSDrop, which included some efforts to port the xbox code to the Windows Research Kernel. I don't believe they were ever able to get anything working, though. I believe the furthest they may have gotten was an XBE loader, but nothing beyond loading the XBE into memory. The issue stems from the fact that Xbox programs run in Kernel mode - if a game crashes, the whole system crashes since control of the OS is handed off entirely to the game program. The 360 did things a bit differently and ran a kernel process underneath all games - hence why the dashboard was accessible from within running games. The Xbox One takes things to another level entirely, and virtualizes every layer of the OS/Dashboard/Game.

The current iteration of the Xbox OS is a far cry from the original implementation. Think of the Xbox OS from 2001 as a sort of Windows PE stuck in textmode, with no drivers for any hardware, while relying on external storage to provide drivers for everything -- including subsystems like directX, sound, and input devices.
Modern Xbox OS is basically Windows 10 with a bunch of specialized components baked in. It provides everything just like a normal Windows installation. Games operate on the XBOne like that of a PC. The Xbox One is a 'stable platform' -- like Apple's macOS is to Apple Mac's.

_________________
Need disks scanned in the USA? I have a Kryoflux, and am willing to help get your disks archived! I also offer xbox and xbox 360 repair and modding services. PM me for details!


Top  Profile  WWW
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Tue Apr 09, 2019 2:04 pm 
Reply with quote
Donator
Offline

Joined
Sat Sep 09, 2006 6:43 am

Posts
773

Favourite OS
Win10/Debian Linux
Let's Talk Xbox Alpha
The first leaked Xbox alpha OS, build 3146.0, from December 2000, boots on normal PC hardware. This is due to the Alpha I and Alpha II hardware being a PC through and through.
The hardware it was designed to run on was a Pentium 3-based system, running on an Intel VC820 motherboard, along with a handful of cards - a PCI USB2 card, an AMR Sound card, and an Intel Ethernet NIC, as well as a Geforce NV15 initially, later a GeForce 3 (NV20).

It's important to note that even at this stage of development, several major changes had been made in the NT kernel - Drivers for video, for instance, aren't provided by driver SYS files. The kernel itself seems to indicate that it does look for, and support two drivers - coverage.sys, and nv4.sys, located in the root drive folder.

Drivers are linked in to dashboard files, and any application built against the kernel. None of this is part of the kernel itself. The kernel itself only provides the Kernel API's and a HAL.

The leaked OS includes a dashboard that supports NV15 cards. The OS itself is a re-designed NT 5.0-ish miniwin-like kernel.
The December 2000 build will run in a virtual machine - to a degree. It bugchecks upon boot, either due to its hardcoded PCI lookups for the addon cards, or the hardcoded lookup for the nVidia GPU.

Later Alpha recoveries provide updated 'drivers' packed into newer dashboard executables. These later recoveries assumed that the user had received an upgrade kit from Microsoft to upgrade the original video card to a GeForce 3 card. As such, the driver packed into these versions is intended to only run on a GeForce 3 GPU, and will crash if anything other than that card is installed in the machine.
Later builds run up through version 1.0.3424 of the XDK, possibly even further, but most of those early builds are currently lost, or in the hands of private collectors who haven't leaked them yet. These later builds also include updated kernels through the use of a bundled XBOXROM.BIN.

Bootloader basics
The December 2000 recovery utilizes a mixture of DOS 7.0 files from Windows 98 SE to boot the system up, then execute xboxldr.com, which sets up the DOS loader of the Xbox kernel, which is stored in XboxBLDR.bin. It is very much like NTLDR, loading in at 0x9000, and loading the kernel from there. The xbox kernel at this point in time was provided as a separate EXE file, in the \XBOX subfolder, named simply xboxkrnl.exe. The December 2000 recovery loads the kernel in a DEBUG mode, allowing some Kernel Debugging via WinDBG - it is, however, missing some crucial bits to make it useful, and appears to only be provided for boot-time debugging on broken hardware (like a virtual machine). The recovery includes an early form of the Xbox Debugger, provided by xbdm.dll. (I've yet to delve into exactly how to utilize these bits for debugging). Later recoveries do not utilize the DOS bootloader at all, and instead, rely on a CDBOOT El-Torito standard bootloader. Comparing NT5 CDBOOT information to these later recoveries' CDBOOT information indicates that they are based on the same code to a degree, with NTLDR replaced by XBOXROM.BIN, along with '/DEBUG' issued as a command line option.

The Xboxldr.com file is a intricate part of the December 2000 recovery's bootloader. It checks a handful of things - are we running within Windows? Is this CPU correct (CPU ID FLAGS)? Is there enough memory in the system (128MB)? It is likely comparable to NTDETECT.COM (I have yet to check this, but some of the information I outline later in this post will likely provide proof that it is a variant of NTDETECT).

XBOXBLDR.bin is the NTLDR in this December 2000 recovery. It's unique to this release of the XDK, and is not included in later builds. However, retail Xbox's do include a form of it as the 1bl, the first bootloader - in the kernel itself.

Is it really NT5?
The Xbox project began in 1998. At the time, NT 5 was still in beta, and a lot of things in the kernel were still in a state of change. Even by December 2000, the kernel was still utilizing code from 1998, not Windows 2000 RTM code. There are strings in the kernel that point to source files that don't exist in later builds. This is key to what lies beneath.
Work on minimizing the kernel for Embedded solutions began with NT4. The Xbox was an embedded platform. It was built between the NT4 embedded line and NT5's beta. As such, it is, at least in the December 2000 version, a combination of NT4 and NT5. It's it's own animal outright. The kernel retains a bit of NT4 here and there, along with features that we would see in NT5. It's not Windows 2000 so much. The inclusion of a multi-executable bootloader instead of the single NTLDR included in NT5 points in that direction, anyway. The security system that Microsoft had in mind required several pieces of boot code to be stored in different locations of the system. Utilizing a modified NTDETECT and NTLDR was a design decision, as some of this code could be located in un-trusted storage, while the core bootloader could be stored elsewhere in the system. They utilized this in the retail machine by storing the kernel in FLASH memory, and the real bootloader in the chipset itself. The FLASH memory contained a fake bootloader that utilized a decryption algorithm to decode the real, hidden bootloader in the chipset, which in turn would decrypt the kernel in FLASH, and boot the rest of the machine.
Instead of re-writing NT5's kernel to support this somewhat older bootloader functionality, they simply took NT4's bootloader, patched it, and used it. This was obvious in the December 2000 recovery, but made less obvious in subsequent releases of the developer tools.

NT4/5/?
The Xbox kernel at this point in time was a reduced functionality, purpose-built kernel. It utilized some NT and Win32 API's from NT5's feature-set, while also adding Xbox API's. XeLoadImage, for instance, provided the loading of Xbox Alpha Executables. These programs are similar to PE-format executables, as they appear to include an 8kb wrapper that sets up the system for executing MZ/PE executables. The included default.xbe, for instance, is really an executable called Recovery.exe, which, according to the compiler-set flags, is a Windows GUI application.

As for those that say "Oh, the Xbox kernel was Windows 2000 based" - no, it really wasn't. It was a ground-up rebuild of the kernel, taking bits and pieces from the Embedded NT4 kernel, and borrowing a few bits from the NT5 kernel, such as the basis for DirectX 8 support, and memory management api's. It was purpose-built, with a set of hardware in mind, such that they wrote an entirely new set of code for the HAL, Memory Management, Filesystem, and various other bits found in the NT code base. It does not retain NT even in name - The December 2000 recovery identifies itself as XBOX OS, with its own set of build numbers, outside of the normal NT build numbers of the era. If anything, it would have been labelled NT 4.5, being a mis-mash of code from both releases of the code base.

Running it
The December 2000 recovery boots in a virtual machine. How far it gets, however -- is not far at all. It expects to see a Pentium 3 CPU, and appears to check CPU Flags to determine if it's running on genuine hardware. It has checks in it for the PCI devices it expects to see, going as far as checking Vendor ID's and Device ID's. Its included default.xbe, the actual Recovery.exe, only includes drivers for the hardware it was destined to run on. This includes an early nVidia driver (which, from my own research, was only officially supporting the NV15 chipset, yet has "XBOX NV 20 Driver" in its strings), which interfaces with the exported API's from the Xbox Kernel. It only seems to run properly in VirtualBox, as attempting to run it in VMWare on an AMD host CPU results in an error message about "Invalid CPU Opcode" (may be related to VMWare's inability to run Win9x on AMD host CPU's). Attempting to run it in anything pre-dating the Pentium 3, like 86Box results in the same error. Attempting to run it in QEMU configured as a Pentium 3 guest, however, also fails.
Booting it in Virtualbox with a Serial port attached as a host pipe to com1 allows some WinDBG kernel debugging output. It bugchecks with a driver error, but due to the limited amount of kernel debugging features included in the kernel itself, provides no indication as to what driver has failed - I don't believe it even gets to running the default.xbe's drivers, but is crashing during HAL initialization.

What's next?
Ideally, I'd like to see this recovery running in a virtual environment. Unfortunately, I'm not a programmer, I can only do so much in patching bits and bytes in hopes of changing Vendor ID's and Device ID's to allow booting in a virtualbox's provided guest hardware.

Then there's the later recoveries. They appear to boot like a NT5 installation - using similar, but not exactly the same CDBOOT data to load the kernel. The XBOXROM.BIN files appear to be compressed, like the later retail kernels were, and are likely also encrypted - or at least they look to be, as loading them into a decompiler provides no strings, unlike the earlier recoveries. The December recovery is unique - everything is right in front of you, debug info is in the kernel, for instance. Unstripped binaries are useful in determining so much, yet, without any of that being obvious, the task becomes impossible for someone like me to decipher. The later recoveries seem to indicate that the XBOXROM.BIN is an executable - it's the first thing to execute after the El-Torito bootsector executes. Ideally, I need to investigate decompressing Microsoft CAB's, as the kernels from after December 2000 likely utilize LZ77 compression to fit them into a >1MB envelope. Once I get past this roadblock (if ever), I should be able to get the resulting Xboxkrnl extracted, and then run into the first roadblock with bugchecks once again.
There is little to go on with these later recoveries - The CDBOOT information indicates that "/DEBUG" is passed as a command line option, but it does not specify a port, nor a baudrate for WinDBG. It's likely that these builds utilized the Xbox Debugger instead of WinDBG - which opens a can of worms. There are legal python-based Xbox Debug tools out there, but they are intended for retail kernels -- I have yet to test them, however, and they may well hold the key to figuring out the later development builds' secrets.

On this topic, the Xbox Dev Wiki has a wealth of useful information -
http://xboxdevwiki.net/Development_Kits
http://xboxdevwiki.net/Xbox_Debug_Monitor
http://xboxdevwiki.net/XBDM_commands_by_version
That last one doesn't list the 3424 build, but it is likely on-par with the later 3521 release.

Next steps...
To conclude this post, my next steps and reasons for digging into this stuff in the first place follows.
There are a handful of debug builds of games that can only be run on the Alpha XDK machines. These machines are far and few, and building one to only use for a handful of debug games is not really cost-effective. Understanding how the machines worked, however, can shed some light into emulating the necessary bits. With emulators like CXBX-Reloaded and XQEMU, it would be advantageous to be able to provide a spec for these tools to offer Alpha support. Being able to run pre-Retail games is the end-all goal for me. There's plenty of pre-release and/or cancelled games left in these unsupported formats to garner a reason to offer support for emulating them.

Next steps include: Finding a way to decompress/decrypt the later recoveries' XBOXROM.BIN, finding a way to patch the December 2000 recovery to run in a virtual machine, and to that degree, run later recoveries in a virtual machine, to at least get kernel debugging working to a point where the knowledge gathered from these early kernel versions can be gathered to emulate them.

_________________
Need disks scanned in the USA? I have a Kryoflux, and am willing to help get your disks archived! I also offer xbox and xbox 360 repair and modding services. PM me for details!


Top  Profile  WWW
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Wed Apr 24, 2019 10:30 pm 
Reply with quote
Donator
Offline

Joined
Sat Sep 09, 2006 6:43 am

Posts
773

Favourite OS
Win10/Debian Linux
I spent some time this past week digging into the recovery code. It appears that all devkits support booting a new bios via CD - including the alpha kits. Yet, I'm unable to get the virtualized version to get far enough to load xboxrom.bin from cd -- or at least it doesn't look like it's loading it, as it should load the newer kernel inside the rom image, but windbg shows only build 3146 as the running kernel. So, I'm working on figuring out exactly how the mechanism works in the alpha xdk to load these roms. Not knowing or having the knowledge of how to determine what the decryption key is for these later bioses is a problem too. I suspect that there is something in the CDBOOT-ish code that includes the decryption routine, so I've got a bit of learning to do in order to isolate this code and then hopefully decompress the later roms. Being able to utilize these later alpha/beta kernels in some way could be beneficial to running early games that were built on this era of Xbox hardware.

_________________
Need disks scanned in the USA? I have a Kryoflux, and am willing to help get your disks archived! I also offer xbox and xbox 360 repair and modding services. PM me for details!


Top  Profile  WWW
 PostPost subject: Re: Xbox OS [Original/360/XBone]        Posted: Sun Apr 28, 2019 4:39 am 
Reply with quote
FTP Access
User avatar
Offline

Joined
Wed Aug 01, 2012 1:27 am

Posts
50

Location
Owensboro, Kentucky

Favourite OS
Longhorn 4008
Im enjoying your updates Jimmsta! Its awesome to see someone dedicated such as you looking deep into this and exploring with it. =)


Top  Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 16 posts ] 




Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  

All views expressed in these forums are those of the author and do not necessarily represent the views of the BetaArchive site owner.

Powered by phpBB® Forum Software © phpBB Group

Copyright © 2006-2019

 

Sitemap | XML | RSS


Affiliate