WinFS

WinFS Beta 1 logo

WinFS (short for Windows File System or Windows Future Storage) was a cancelled storage subsystem that allowed users to search for items based on their metadata, regardless of the item's type of file and the application in which it was created. This in turn would make searching for content independent of format easier.

WinFS was billed as one of the pillars of Windows Longhorn. However, Microsoft announced on 27 August 2004 that WinFS would be in beta testing when Longhorn becomes available, and would be delivered after the Longhorn release. On 29 August 2005, Microsoft (through Vijau Bangaru, a Program Manager on the WinFS product team) announced the release of WinFS Beta 1 for Windows XP Service Pack 2. Regular beta and CTP releases were planned. However, on 23 June 2006, Microsoft (through Quentin Clark, who ran the Program Management Team for WinFS) announced that there would no longer be a separate WinFS release. Instead, some of its parts were included in ADO.NET and Microsoft SQL Server.

WinFS was based on SQL Server 2005, ran on top of NTFS, and used metadata of files to improve organizing, searching and sharing of information. It also stored structured data such as contacts and calendars. WinFS replaced the traditional file system hierarchy of files and folders with items and associations. Items could either be backed by files (for example, images and documents) or not (for example, contacts and e-mail). Associations established relationships between different types of data (items), which then allowed users to organize information based on how they used them.

Development

Screenshot of Microsoft Cairo/4.0.1175.1. Note that local disk D:\ is under an OFS partition

WinFS is ultimately traceable to the Object File System (OFS), an integrated store, which Microsoft planned to be part of Microsoft Cairo. The idea behind OFS "was to make sense out of all the bytes of data in the filesystem in a human readable and searchable way, organized as a collection of objects with properties all part of a strongly typed schema." OFS was a relational file system, "which would help manage local and distributed objects using an active storage paradigm" and "would have removed the need to understand where data resided", both locally and remotely. One build of Cairo, 1175.1, allowed OFS partitions (see screenshot). However, it is not clear if the features described for OFS are present.

Cairo fell apart in the mid-1990s. OFS merged into Storage+, which Paul Thurrott wrote, "Microsoft will turn its attention to" after Windows 2000 ships. Storage+ would replace the NTFS file system and with a relational, object-oriented file system based on SQL Server 8.0, turning the file system into a relational database, speeding searches and providing a higher level of granularity for those searches Storage+ later became the Relational File System, which Microsoft intended to but failed to ship with SQL Server 2000.

Storage+ (at this point "a new SQL Server .NET-based file system") was then slated for Blackcomb, but by 2002, Thurrott verified that it would ship with Longhorn. Storage+ later became the Relational File System, and Microsoft intended to but failed to ship it with SQL Server 2000. By 2003, Storage+ was called WinFS, would include technology from SQL Server "Yukon", and would not be a file system but a service on top of NTFS.

During PDC 2003, Quentin Clark, director of program management, WinFS, explained:

WinFS is...

• All end-to-end data lives in Longhorn
• New user experience in the Longhorn shell
• A trustworthy place to store data
• Data model built on relational database technology
• File system capabilities built on NTFS
• Everyday info - domain specific schemas - Longhorn will come with set of schemas Services that make data interactive

WinFS is not...

• Abandoning legacy applications
• Forcing the world to rewrite everything
• Complete yet
  • PDC is good preview of model, feedback welcome
  • Fundamental model complete for Beta 1
  • Platform and UE complete in Beta 2
• Just a database - MS has set of relational assets, WinFS is pulling techs from NTFS, SP, and CLR too

Longhorn build 4051, which was distributed during PDC 2003, contained WinFS. However, it suffered from huge performance issues. However, on 27 August 2004, Microsoft announced that WinFS would be in beta testing when Longhorn becomes available, and would be delivered after the Longhorn release. This meant that WinFS would be removed from Longhorn and would ship at a later date.

On 29 August 2005, Microsoft quietly made WinFS Beta 1 available to MSDN subscribers, but announced it officially through the WinFS Team Blog. Beta 1 required Windows XP Service Pack 2, and .NET Framework 2.0.50215.322. The WinFS API was included in the System.Storage namespace.

During PDC 2005, Microsoft released a video presentation, named "IWish", showing how applications would advantage of a unified type system. The concepts shown in the video ranged from applications using the relationships of items to dynamically offer filtering options to applications grouping multiple related data types and rendering them in a unified presentation. Steve De Mar, the Product Designer for the WinFS team, explained the goals of IWish:

Our goal was to peak the curiosity of you, the developers, to get you thinking about what could really be built on WinFS. We wanted you to see how WinFS enables a totally new class of user experiences. The UIs in the video are all based on some of the great prototype applications that have already been developed for the WinFS platform. Although we cast them in a more futuristic visual “theme” to fit the style of the video, the functionality we showed exists in the prototypes that either we’ve developed in house or we’ve seen others develop on WinFS. A number of these were demo’d at the PDC, and we've more on the way which you should be hearing about shortly.

We hope the video inspires you to bring to life cool new innovative applications for WinFS!

In addition, during PDC 2005, Shishir Mehrota of WinFS Product Planning gave an overview of WinFS, Program Manager Ramesh Nagarajan gave a presentation on "“WinFS” Future Directions: Building Data-Centric Applications Using Windows Presentation Foundation “Avalon” and Windows Forms", and Samuel Drucker, Development Manager, gave a presentation on WinFS and ADO.NET (on how ADO.NET is evolving to incorporate some of the WinFS aspects) and WinFS Programming with LINQ.

Beta 1 Refresh was released to MSDN subscribers on 1 December 2005 to be compatible with the final release of .NET Framework 2.0.

WinFS Beta 2 was planned for some time later in 2006. Four WinFS sessions were planned for Tech Ed 2006. But on 23 June 2006, Quentin Clark announced through the WinFS Team Blog, "With most of our effort now working towards productizing mature aspects of the WinFS project into SQL and ADO.NET, we do not need to deliver a separate WinFS offering."

Many of the principal features Microsoft intended to provide with WinFS included a pane for metadata property editing, breadcrumb-based property navigation, filtering or stacking items over properties, incremental search, and saved searches; features were incorporated in Windows Vista. Query composition, a WinFS feature that allowed users to perform additional searches that reuse the results of a previous query, was later incorporated in Windows Vista.

Examples of uses of the technology are the object-relational mapping components into ADO.NET Entity Framework; support for unstructured data, adminless mode of operation, support for file system objects via the FILESTREAM data type, and hierarchical data in SQL Server 2008, then codenamed Katmai, as well as integration with Win32 APIs and Windows Shell and support for traversal of hierarchies by traversing relationships into later releases of Microsoft SQL Server and the synchronization components into Microsoft Sync Framework.

In 2013, Bill Gates cited WinFS as his greatest disappointment at Microsoft:

We had a rich database as the client/cloud store that was part of a Windows release that was before its time. This is an idea that will [re-emerge] since your cloud store will be rich with schema rather than just a bunch of files and the client will be a partial replica of it with rich schema understanding.

Limitations of traditional file systems

Traditional file systems, including NTFS which is used in modern versions of Microsoft Windows, store data only as a stream of bytes packed in a file. The Microsoft WinFS SDK Beta 1 Documentation explained the limitations of this approach:

In a file system, data is stored as one or more byte streams packed in a file. This flat data structure does not make the information easily identifiable. The only viable way to identify it is to use the file name and its directory path, that is, a folder hierarchy. Although a file may contain many kinds of information, it is the smallest unit of data that can be moved around and operated on. For example, to read a paragraph of a document, you need to retrieve the entire file and open it in a document viewer.

As Richard Grimes explained:

The perennial problem with data is that you always have too much of it and it is always too difficult to find what you want. This problem will get far worse in the near future when typical machines will have hard disks measured in terabytes. The bigger the hard disk, the more data you will want to store, and the more difficult it will be to find a particular item. Finding a piece of data depends on identifying the specific item, which means that the item has to be categorized.

For example, imagine that you have several thousand digital photographs. Typically, your camera and desktop software will catalog this data by naming the photos in sequence and maybe place these in a folder according to the date that the photos were taken. Searching for a particular photo in this scheme involves remembering the date and photo number. If you do not know either of these things, you will have to resort to looking through thumbnails of the photos. Of course, if you have the patience, you could name each photograph as you download it to your desktop. A coarse-grained system, like the file and folder system in Win32®, is not flexible enough to allow you to store all the data you need.

Desktop software may fill this gap, allowing you to build a catalog that stores other descriptive properties like camera settings, light condition, and a general description of each picture. Such software, of course, is an implementation of a database, but the important point is that it acts as a supplement to the file system. You could argue that these descriptions of your photos—metadata—are actually part of the photograph's object. Digital photographs hold other information within the image file's EXIF (Exchangeable Image File Format) header. This header contains information obtained when the photo was taken. To include this metadata in a search would require a query engine to extract this information from each candidate photo and compare it with the query condition. A sophisticated photograph organizer could extract the EXIF information when the photograph is transferred to the desktop and include it as additional metadata in the database.

Grimes pointed out that one of the monumental problems organizations face is aggregating information that is stored in disparate formats. Knowledge workers have long wanted to be able to search for content independent of format. Sean Grimaldi of the WinFS team called the existence of many formats with different expectations and behaviors a "divide-and-conquer approach" in dealing with data. Grimaldi extensively discussed the negative effects of this approach and summarizes:

The data is divided into file, database, and other data. The negatives are that programmers have to recreate data types, bridge between data types, and rewrite behaviors repeatedly. If this seems normal to you, it may be because you have operated in this fractured data environment for years and have come to expect nothing more.

The Microsoft WinFS SDK Beta 1 Documentation described the existence of proprietary file formats as "separate islands":

Proprietary file formats tend to aggregate data into separate islands, creating barriers for data sharing between applications. For example, an address book created by an e-mail system, such as Microsoft Outlook, is not directly usable by an appointment-setting application unless the latter adopts the same file format or knows how to translate the format from one to another.

Shawn Wildermuth pointed out what this system means for developers:

Data integration solutions must first try to make sense of all the different places applications can store information. Let's look at a typical example:

I am writing an application, I need access to the user's Contact List, Documents folder, and Music Folder. To access each of these data stores, I have to deal with three different APIs to get that data (MAPI to access Contacts, System.IO for the Documents folder, and the Windows Media Player SDK to find all the music for a user). It would be better if we had a consistent API for users' data.

But that is not the worst part of the problem. If my application wants to tie its data and these external pieces together, it will need code that not only establishes the relationship between the objects, but deals with changes to that data and handles data synchronization problems such as orphaned data.

Using common file formats is a workaround to this problem but not a universal solution; there is no guarantee that all applications will use the format. Other solutions used by corporations are database systems and XML and XML web services, as Thomas Rizzo, director in the Microsoft SQL Server group, explained:

In the beginning, computers could not handle the complexities of modeling and automating many business processes. However, with faster computers, better programming languages, and better data technologies, automation of business processes became a way for corporations to streamline their operations, especially with the advent of enterprise resource planning (ERP), customer relationship management (CRM), sales force automation (SFA), and other types of enterprise applications. With these new types of applications, data suddenly became more complex—but at the same time, data had better structure that was more useful for businesses. Rather than storing opaque binary data or simplistic data models, database systems could relate different types of complex data together. For example, ERP systems understand and can model the complexities of general ledgers, human resources, and sales systems. Since standardized schemas described the data in the system, corporations could ask interesting questions about their business and query their systems for that information.

However, working across systems is still a problem in the industry. Not only is it hard to integrate systems within a corporation, but integrating systems across corporations is even harder. This has sparked the growth of enterprise application integration (EAI) vendors who help customers navigate the sea of schematized data across heterogeneous systems. Recognizing this integration problem, the industry has looked to XML and XML Web services as a way to help companies work across multiple systems and multiple organizations. These newer standards help solve data integration problems and are helping to democratize data throughout and across organizations.

Traditional file systems also provide only a single way of organizing the files, namely via directories and file names. Vijay Bangaru of the WinFS team illustrated the problems with this:

If you’re like me, you probably have tons of digital pictures on your computer. My photos are currently organized in a simple hierarchy. In “My Pictures”, I have a bunch of folders that are named something like, “2004-11 Thanksgiving”. Within this folder are all my photos from that holiday. I’m sure you can already see the limitations in this structure.

My latest challenge in actually using my pictures came the other day. My grandparents wanted me to send them some pictures of my fiancée and me. Wouldn’t it be really cool if I could just tell my computer, “Show me all the pictures that have Crissy and me in them”. Instead, I had to scan through my folders to see what events we were both at and copy all the pictures with both of us in them to a temp folder. Then I needed to go through the temp folder to pick which ones to send. Of course, I didn’t remember to delete the temp folder; so it’s eating up drive space. Good thing hard drives are so cheap nowadays.

A traditional file system can retrieve and search data based only on the directory path:

If the user knows the directory path to the file, searching is simple. But if she does not know where the file is, she must walk down from the root directory of a file system—for example, by selecting Search from the Start menu—to traverse the available directory paths until she finds the file. When there are many files on a computer, the search can take a long time. In the NTFS file system and other traditional file systems, the directory path is more or less the only way to identify a file. But users tend to think of data more naturally in terms of its types and attributes—that is, the structure or shape of the data: a document or an e-mail message, its title or subject, the author or sender. How the data is stored is generally of little or no interest. Therefore, reliance on paths is an inherent limitation of the tree-based structure of traditional file systems.

A user may put a file into multiple directories, but this wastes disk space. Also, when a copy is changed, the other copies no longer represent the same data as the changed copy. Hard links that can be created to represent a file by multiple paths are also difficult to manage: links will break when a file is renamed.

Overview

The core concept of WinFS was that by storing data in a cohesive data store, not just a file system, it would have allowed systems and users to store rich metadata about a myriad of objects and to relate these objects. Compared to a file in a traditional file system, a "WinFS" item would be the smallest unit of data that could be moved around and operated on by different users or applications.

The Microsoft WinFS SDK Beta 1 Documentation explained that WinFS would have provided comprehensive object-relational mappings to let users view and work with their data as an item on a Windows computer. With items, a user or an application would have found it more logical or natural to view and use data. This should have made organizing and querying data, especially complex data, less burdensome. The documentation provided as an example:

For example, consider an e-mail message containing two photos as attachments. In the tabular form, this is most likely modeled as three rows in two tables. One table stores all the e-mail messages and the other stores all the photos. To get the message, a user or application must query two tables and perform the so-called UNION operations on the tables. In the itemized form, the attachment is part of the item. Querying the e-mail message item will return the attached photos as well.

WinFS would have served all kinds of data, whether structured (as in tables/rows or items), semi-structured (as in an XML file), or unstructured (as in a flat byte stream). In the WinFS Beta 1 release, it did so through a new file system driver (RsFxDrv.sys).

The Microsoft WinFS SDK Beta 1 Documentation and the IWish video summarized the benefits of WinFS integration into four points: Unify, Organize, Explore, and Innovate.

Unify

WinFS would have allowed data sharing between applications, as Grimaldi explained:

Because applications can build on WinFS-defined data types, applications can share data—assuming the correct permissions. This is good for you as a programmer, as well as for the end user of your application.

As a programmer, you can reuse data the end user already has in the WinFS store. For example, assume your application presents advertisements to a user of events they could attend, like a bike race or a rodeo.

Before presenting an event to the user, your application can check the user's calendar and display the event differently if the user already has a meeting scheduled that would conflict with the event, such as making the border of the invitation red.

You can also populate data into the WinFS store to be used by other programs. For example, if the end user accepts the invitation to the bicycle race, their personal information manager (PIM) application, such as their e-mail client, could show that date as reserved in their calendar. This also gives the end user a rich user experience in both applications.

WinFS would have placed data from multiple applications into one central store. Bangaru showed the advantages of this:

With WinFS, I can move all my contact information into one central store. My applications can interact with this information and I can synchronize it wherever I want to. The obvious benefit is that I can go to one place reliably for the most up to date information. Now, I could easily add whoever I wanted to an Evite for a Halloween party. No more need for typing in email addresses or importing all my contacts (into yet another silo). When I order flowers for mom on her birthday, it’s just one click for me to fill out the information for the recipient.

In addition to being a central data store, WinFS can be your synchronization hub. I can sync between WinFS Stores and I can sync changes back and forth with legacy stores (like Outlook, PDAs, and cell phones). Now, I can have my latest contact info on my phone, make changes on the road, and sync that information back to WinFS. This, of course, enables some great offline experiences. When I’m doing work on vacation, I don’t need to worry about manually synchronizing my data with my desktop machine once I’m home. Now all my applications will behave like Outlook 11’s awesome offline mode.

Of course, this applies to all data; not just contacts. Now, all my calendar apps will show my Evite invites. My photos are now associated with contacts and events, so I can easily say, “Mail these photos to everyone that was at the event”. When you start adding email, RSS feeds, documents, music and everything else to this, it can get pretty exciting.

Organize

Data stored and managed by the system are instances of the data type recognized by the WinFS runtime. The data are structured by means of properties. For example, an instance of a résumé type will surface the data by exposing properties, such as Name, Educational Qualification, Experience. Each property may be a simple type (strings, integers, dates) or complex types (contacts). Different data types expose different properties.

Besides that, WinFS also allows different data instances to be related together. Relationships are also exposed as properties; for example if a document is related to a contact by a Created By relationship, then the document will have a Created By property. When it is accessed, the relationship is traversed and the related data returned. By following the relations, all related data can be reached. Bangaru illustrated how data could be related, and how related data could be shown:

With WinFS, I can relate my contacts to my photos, so I can easily query for all the pictures that are related to a set a contacts. A photo app built on WinFS could display all the pictures with Crissy and me in them. Then, I could select the good ones to email or print. If you’ve been playing with the Beta 1 bits, you know how easy it is to write an app with this behavior.

Plus, I can now easily check to see what photos I’ve already sent, so I don’t send the same ones again. How? Tell the computer to “show me the photos with both of us that I’ve sent to my grandparents.” All these containment relationships and associations would already be in my store.

Again, this applies to much more than pictures. I can associate a document with a meeting, or music and contacts to a party invite. And these aren’t keywords added to a file, these are associations between real items. All of these organization constructs are just waiting to be exploited by a WinFS app.

Also, all these types are extensible. So for my wedding guest list, I can extend the Contact type to have properties for “Sent ‘Save the Date’ card”, “Sent invitation”, “RSVP Received”, “# of Guests”, “Vegetarian meal?” and so on. That could make things significantly easier. Good thing I can do this with the Beta bits. If we had to wait for WinFS RTM to plan the wedding, I think Crissy would make sure that I wouldn’t be able to blog anymore. :-)

Explore

WinFS natively recognized different types of data by defining common item types, such as audio record, meeting, contact, person, document, photo, message, video clip. This would have solved the problem of querying the following data:

You could write code that shows a picture of a person, his role, and his contact information for everyone attending a given meeting and every document that they contributed to that you have not already read, and on a personal note, if they have already given you some money for the charity bicycle race you are participating in next month. Today, I have all this data available to my computer, but no feasible way to query it without WinFS. I plainly can't get at a lot of my data.

Bangaru explained how one of the mockup applications showed in the IWish video would use WinFS:

WinFS gives you the ability to use relationships to find and display your data. In the PDC video, we showed an app that takes a range of dates and displays the location for your meetings on the map. As it draws the best route to drive between all your meetings, it adds icons that represent your contacts and important customers at their location on the map. Now, as you visualize your schedule, you can see who is close by and set up lunch.

Let’s walk through how WinFS made this possible. The app is able to query for all your appointments in a certain date range. It displays each appointment in the proper place on the map by using location of the meeting. Today it would be tough for a map application to do this, because your appointments are in a different application or maybe even more than one application. Assuming you can access the meetings programmatically, accessing the location may be hard. Your calendar app probably just has a field called “Location” that can contain an arbitrary string and would be tough to parse correctly. With WinFS, all your appointment information is in a single store with an API that let’s any application access the data uniformly.

Next, the application finds contacts and customers with high priority issues that are located close to where you will be and displays them on the map, by using the address of the contact. If you select a contact, the app can display some information for the person (a photo and contact info for example). From there, you can send them a mail to setup a lunch while you’re in town. Again this would be tough to do today; your contact information is probably in a silo that your map application can’t get to. Plus, chances are, your customers with high priority issues are tracked in a completely different application silo.

Innovate

Bangaru explained how WinFS would have done away with proprietary file formats:

WinFS is built on a true relational store that all applications can leverage. Today, tons of popular apps lock their data into proprietary stores. This includes photo apps like Photoshop album, music apps like WMP and iTunes, financial apps like Quicken and Money, email apps like Outlook, and so on. By using WinFS, developers can concentrate on building rich applications without worrying about creating a storage silo. After all, WinFS can give applications a place for all their structured, semi-structured, and unstructured data.

In addition, WinFS will provide services like synchronization, notifications, rules, and backup/restore. The services provided by WinFS lead to a zero-admin and extremely reliable experience.

Synchronization was explained earlier under Unify.

Kati Dimitrova, a developer on the WinFS API team, explained how WinFS Rules work through a scenario involving Mary, a recruiter trying to manage and track candidates throughout her day across multiple applications:

First and foremost, WinFS rules are persistable units of logic. In this scenario, Mary creates the query “candidates that are interviewing today” through her WinFS-enabled recruiting application and persists it. But how is the query persisted? As a WinFS item. In fact, all WinFS rules, including queries, are items. This has many advantages. To start with, rules can be created, manipulated, stored and secured in the same manner as all other items.

Further, WinFS rules are about discovery and sharing of persisted logic among applications. Because rules are WinFS items, they can also be queried for, discovered and shared among applications just like all other WinFS items. In this scenario, Mary can easily find and use the query “candidates that are interviewing today” from her email application even though she created it through her recruiting application.

Finally, WinFS rules are about re-usability and composition. Not only can Mary find the query she created in her recruiting application, she can re-use it in her email application to create a new query “mail related to candidates that are interviewing today” that refers to the original query. Note that composition of rules is true composition; if Mary changes the logic that defines the query “candidates that are interviewing today,” those changes will affect the results of the query “mail related to candidates that are interviewing today.” Bangaru explained two levels of backup that WinFS would have included. Item level backup would have allowed a user to revert a change to an item. Store level backup would have covered the situation of a broken hard drive and a user needing all his files back.

Data storage

Architecture

What is WinFS?

This section discusses WinFS as it worked under WinFS Beta 1, not under any Longhorn pre-reset build. All documentation therefore should refer to the WinFS Beta 1 time frame, PDC 2005 included.

WinFS used a subset of SQL Server 2005 technology, modified for file storage. WinFS stored filestreams as ordinary NTFS files with a GUID as file name under the "System Volume Information" folder. Meanwhile, metadata is stored in a SQL Server database.

WinFS ran as a service that runs three processes:

1. WinFS.exe, which hosts relational datastore
2. WinFSSearch.exe, which hosts the indexing and querying engine
3. WinFPM.exe (WinFS File Promotion Manager), which interfaces with the underlying file-system

It allows programmatic access to its features via a set of .NET Framework APIs. These enable applications to define custom-made data types, define relationships among data, store and retrieve information, and allow advanced searches. The applications can then aggregate the data and present the aggregated data to the user.

Data store

WinFS stores data in relational stores, which are exposed as virtual locations called stores. A WinFS store is a common repository where any application can store data along with its metadata, relationships and schema. WinFS runtime can apply certain relationships itself; for example, if the values of the subject property of a picture and the name property of a contact are same, then WinFS can relate the contact with the picture. Relations can also be specified by other applications or the user.

WinFS provides unified storage, but stops short of defining the format that is to be stored in the data stores. Instead it supports writing data in application-specific formats. But applications must provide a schema that defines how the file format should be interpreted. For example, a schema could be added to allow WinFS to understand how to read and thus be able to search and analyze, (say) a PDF file. By using the schema, any application can read data from any other application, and this also allows different applications to write in each other's format by sharing the schema.

Multiple WinFS stores can be created on a single machine. This allows different classes of data to be kept segregated; for example, official documents and personal documents can be kept in different stores. WinFS, by default, provides only one store, named "DefaultStore". WinFS stores are exposed as shell objects, akin to Virtual folders, which dynamically generate a list of all items present in the store and present them in a folder view. The shell object also allows searching information in the datastore.

A data unit that has to be stored in a WinFS store is called a WinFS Item. A WinFS item, along with the core data item, also contains information on how the data item is related to other data. This Relationship is stored in terms of logical links. Links specify which other data items the current item is related with. Put in other words, links specify the relationship of the data with other data items. Links are physically stored using a link identifier, which specifies the name and intent of the relationship, such as type of or consists of. The link identifier is stored as an attribute of the data item. All the objects that have the same link ID are considered to be related. An XML schema, defining the structure of the data items that will be stored in WinFS, must be supplied to the WinFS runtime beforehand. In Beta 1 of WinFS, the schema assembly had to be added to the GAC before it could be used.

Data model

WinFS models data using the data items, along with their relationships, extensions and rules governing its usage. WinFS needs to understand the type and structure of the data items, so that the information stored in the data item can be made available to any application that requests it. This is done by the use of schemas. For every type of data item that is to be stored in WinFS, a corresponding schema needs to be provided to define the type, structure and associations of the data. These schemas are defined using XML.

Predefined WinFS schemas include schemas for documents, e-mail, appointments, tasks, media, audio, video, and also includes system schemas that include configuration, programs, and other system-related data. Custom schemas can be defined on a per-application basis, in situations where an application wants to store its data in WinFS, but not share the structure of that data with other applications, or they can be made available across the system.

Type system

File:WinFS1.svg

WinFS Type Hierarchy

The most important difference between a file system and WinFS is that WinFS knows the type of each data item that it stores. And the type specifies the properties of the data item. The WinFS type system is closely associated with the .NET framework's concept of classes and inheritance. A new type can be created by extending and nesting any predefined types.

WinFS provides four predefined base types – Items, Relationships, ScalarTypes and NestedTypes. An Item is the fundamental data object which can be stored, and a Relationship is the relation or link between two data items. Since all WinFS items must have a type, the type of item stored defines its properties. The properties of an Item may be a ScalarType, which defines the smallest unit of information a property can have, or a NestedType, which is a collection of more than one ScalarTypes and/or NestedTypes. All WinFS types are made available as .NET CLR classes.

Any object represented as a data unit, such as contact, image, video, document etc., can be stored in a WinFS store as a specialization of the Item type. By default, WinFS provides Item types for Files, Contact, Documents, Pictures, Audio, Video, Calendar, and Messages. The File Item can store any generic data, which is stored in file systems as files. But unless an advanced schema is provided for the file, by defining it to be a specialized Item, WinFS will not be able to access its data. Such a file Item can only support being related to other Items.

File:WinFS2.svg

Defining a new Type

A developer can extend any of these types, or the base type Item, to provide a type for his custom data. The data contained in an Item is defined in terms of properties, or fields that hold the actual data. For example, an Item Contact may have a field Name that is a ScalarType, and one field Address, a NestedType, which is further composed of two ScalarTypes. To define this type, the base class Item is extended and the necessary fields are added to the class. A NestedType field can be defined as another class that contains the two ScalarType fields. Once the type is defined, a schema has to be defined, which denotes the primitive type of each field, for example, the Name field is a String, the Address field is a custom defined Address class, both the fields of which are Strings. Other primitive types that WinFS supports are Integer, Byte, Decimal, Float, Double, Boolean and DateTime, among others. The schema will also define which fields are mandatory and which are optional. The Contact Item defined in this way will be used to store information regarding the Contact, by populating the properties field and storing it. Only those fields marked as mandatory needs to be filled up during initial save. Other fields may be populated later by the user, or not populated at all. If more properties fields, such as last conversed date, need to be added, this type can be extended to accommodate them. Item types for other data can be defined similarly.

File:WinFS3.svg

Relationships

WinFS creates tables for all defined Items. All the fields defined for the Item form the columns of the table and all instances of the Item are stored as rows in the table for the respective Items. Whenever some field in the table refers to data in some other table, it is considered a relationship. The schema of the relationship specifies which tables are involved and what the kind and name of the relationship is. The WinFS runtime manages the relationship schemas. All Items are exposed as .NET CLR objects, with a uniform interface providing access to the data stored in the fields. Thus any application can retrieve object of any Item type and can use the data in the object, without being aware of the physical structure the data was stored in.

WinFS types are exposed as .NET classes, which can be instantiated as .NET objects. Data are stored in these type instances by setting their properties. Once done, they are persisted into the WinFS store. A WinFS store is accessed using an ItemContext class (see Data retrieval section for details). ItemContext allows transactional access to the WinFS store; i.e. all the operations since binding an ItemContext object to a store till it is closed either all succeed or are all rolled back. As changes are made to the data, they are not written to the disc; rather they are written to an in-memory log. Only when the connection is closed are the changes written to the disc in a batch. This helps to optimize disc I/O. The following code snippet, written in C#, creates a contact and stores it in a WinFS store.

<syntaxhighlight lang="csharp">

//Connect to the default WinFS store
using(ItemContext ic = ItemContext.Open())
{
    //Create the contact and set the data in appropriate properties
    ContactEAddress contact = new ContactEAddress() {
        Name = new PersonName() {                      // Name is a ComplexType
            Displayname = "Doe, John",
            FirstName = "John",
            LastName = "Doe"
        },
        TelephoneNumber = new TelephoneNumber() {     // Telephone number is a ComplexType
            Country = CountryCode.Antarctica,
            Areacode = 4567,
            Number = 9876543210
        },
        Age = 111                                     // Age is a SimpleType
   };

   //Add the object to the user's personal folder.
   //This relates the item with the Folder pseudo-type, for backward
   //compatibility, as this lets the item to be accessed in a folder 
   //hierarchy for apps which are not WinFS native.
   Folder containingFolder = UserDataFolder.FindMyPersonalFolder();
   containingFolder.OutFolderMemberRelationship.AddItem(ic, contact);

   //Find a document and relate with the document. Searching begins by creating an
   //ItemSearcher object. Each WinFS type object contains a GetSearcher() method
   //that generates an ItemSearcher object which searches documents of that type.
   using (ItemSearcher searcher = Document.GetSearcher(ic))
   {
        Document d = searcher.Find(@"Title = 'Some Particular Document'");
        d.OutAuthoringRelationship.AddItem(ic, contact);
   }
   //Since only one document is to be found, the ItemContext.FindOne() method
   //could be used as well.

   //Find a picture and relate with it
   using (ItemSearcher searcher = Picture.GetSearcher(ic))
   {
        Picture p = searcher.Find(@"Occasion = 'Graduation' and Sequence = '3'");
        p.OutSubjectRelationship.AddItem(ic, contact);
   }

   //Persist to the store and close the reference to the store
   ic.Update();
}

</syntaxhighlight>

Relationships

A datum can be related to one more item, giving rise to a one-to-one relationship, or with more than one items, resulting in a one-to-many relationship. The related items, in turn, may be related to other data items as well, resulting in a network of relationships, which is called a many-to-many relationship. Creating a relationship between two Items creates another field in the data of the Items concerned which refer the row in the other Item's table where the related object is stored.

File:WinFS4.svg

WinFS Relationships

In WinFS, a Relationship is an instance of the base type Relationship, which is extended to signify a specialization of a relation. A Relationship is a mapping between two items, a Source and a Target. The source has an Outgoing Relationship, whereas the target gets an Incoming Relationship. WinFS provides three types of primitive relationships – Holding Relationship, Reference Relationship and Embedding Relationship. Any custom relationship between two data types are instances of these relationship types.

• Holding Relationships specifies ownership and lifetime (which defines how long the relationship is valid) of the Target Item. For example, the Relationship between a folder and a file, and between an Employee and his Salary record, is a Holding Relationship – the latter is to be removed when the former is removed. A Target Item can be a part of more than one Holding Relationships. In such a case, it is to be removed when all the Source Items are removed.
• Reference Relationships provide linkage between two Items, but do not have any lifetime associated, i.e., each Item will continue to be stored even without the other.
• Embedding Relationships give order to the two Items that are linked by the Relationship, such as the Relationship between a Parent Item and a Child Item.

Relationships between two Items can either be set programmatically by the application creating the data, or the user can use the WinFS Item Browser to manually relate the Items. A WinFS item browser can also graphically display the items and how they are related, to enable the user to know how their data are organized.

RAV

WinFS supports creating Rich Application Views (RAV) by aggregating different data in a virtual table format. Unlike database view, where each individual element can only be a scalar value, RAVs can have complex Items or even collections of Items. The actual data can be across multiple data types or instances and can even be retrieved by traversing relationships. RAVs are intrinsically paged (dividing the entire set of data into smaller pages containing disconnected subsets of the data) by the WinFS runtime. The page size is defined during creation of the view and the WinFS API exposes methods to iterate over the pages. RAVs also supports modification of the view according to different grouping parameters. Views can also be queried against.

Access control

Even though all data are shared, everything is not equally accessible. WinFS uses the Windows authentication system to provide two data protection mechanisms. First, there is share-level security that controls access to your WinFS share. Second, there is item level security that supports NT compatible security descriptors. The process accessing the item must have enough privileges to access it. Also in Vista there is the concept of "integrity level" for an application. Higher integrity data cannot be accessed by a lower integrity process.

Data retrieval

File:WinFSFlowChart.svg

Flowchart for creating, searching and updating WinFS data instances

The primary mode of data retrieval from a WinFS store is querying the WinFS store according to some criteria, which returns an enumerable set of items matching the criteria. The criteria for the query is specified using the OPath query language. The returned data are made available as instances of the type schemas, conforming to the .NET object model. The data in them can be accessed by accessing the properties of individual objects.

Relations are also exposed as properties. Each WinFS Item has two properties, named IncomingRelationships and OutgoingRelationships, which provide access to the set of relationship instances the item participates in. The other item which participates in one relationship instance can be reached through the proper relationship instance.

The fact that the data can be accessed using its description, rather than location, can be used to provide end-user organizational capabilities without limiting to the hierarchical organization as used in file-systems. In a file system, each file or folder is contained in only one folder. But WinFS Items can participate in any number of holding relationships, that too with any other items. As such, end users are not limited to only file/folder organization. Rather, a contact can become a container for documents; a picture a container for contacts and so on. For legacy compatibility, WinFS includes a pseudo-type called Folder, which is present only to participate in holding relationships and emulate file/folder organization. Since any WinFS Item can be related with more than one Folder item, from an end user perspective, an item can reside in multiple folders without duplicating the actual data. Applications can also analyze the relationship graphs to present various filters. For example, an email application can analyze the related contacts and the relationships of the contacts with restaurant bills and dynamically generate filters like "Emails sent to people I had lunch with".

Searches

The WinFS API provides a class called the ItemContext class, which is bound to a WinFS store. The ItemContext object can be used to scope the search to the entire store or a subset of it. It also provides transactional access to the store. An object of this class can then spawn an ItemSearcher object which then takes the type (an object representing the type) of the item to be retrieved or the relationship and the OPath query string representing the criteria for the search. A set of all matches is returned, which can then be bound to a UI widget for displaying en masse or enumerating individually. The properties items can also be modified and then stored back to the data store to update the data. The ItemContext object is closed (which marks the end of association of the object with the store) when the queries are made or changes merged into the store.

Related items can also be accessed through the items. The IncomingRelationships and OutgoingRelationships properties give access to all the set of relationship instances, typed to the name of the relationship. These relationship objects expose the other item via a property. So, for example, if a picture is related to a picture, it can be accessed by traversing the relationship as:

<syntaxhighlight lang="csharp">

ContactsCollection contacts = picture.OutgoingRelationships.Cast(typeof(Contact)).Value;
//This retrieves the collection of all outgoing relationships from a picture object
//and filters down the contacts reachable from them and retrieves its value.

//Or the relationship can be statically specified as
ContactsCollection contacts = picture.OutgoingRelationships.OutContactRelationship.Contact;

</syntaxhighlight>

Template:Anchor An OPath query string allows to express the parameters that will be queried for to be specified using Item properties, embedded Items as well as Relationships. It can specify a single search condition, such as "title = Something'", or a compound condition such as "title = 'Title 1' || title = 'Title 2' && author = 'Someone'". These boolean and relational operations can be specified using C# like &&, ||, =, != operators as well as their English-like equivalent like EQUAL, NOT EQUAL. SQL like operators such as LIKE, GROUP BY and ORDER BY are also supported, as are wildcard conditions. So, "title LIKE 'any*'" is a valid query string. These operators can be used to execute complex searches such as <syntaxhighlight lang="csharp">

using (ItemContext ic = ItemContext.Open() )
{
    //Searching begins by creating a ItemSearcher object. The searcher is created from a
    //relationship instance because the contacts being searched for are in relation. The
    //first parameter defines the scope of the search. An ItemContext as the scope means
    //the entire store is to be searched. Scope can be limited to a set of Items which may
    //be in a holding relationship with the contacts. In that case, the set is passed as
    //the scope of the search.
    ItemSearcher searcher = OutContactRelationship.GetTargetSearcher(ic, typeof(Contact));
    ContactCollection contacts = searcher.FindAll("OutContactRelationship.Contact.Name LIKE 'A*'");
}

</syntaxhighlight> The above code snippet creates an ItemSearcher object that searches on the OutContactRelationship instance that relates pictures and contacts, in effect searching all pictures related with a contact. It then runs the query Name LIKE 'A*'" on all contacts reachable through OutContactRelationship, returning the list of "contacts whose names start with A and whose pictures I have". Similarly, more relationships could be taken into account to further narrow down the results. Further, a natural language query processor, which parses query in natural language and creates a well-formed OPath query string to search via proper relationships, can allow users to make searches such as "find the name of the wine I had with person X last month", provided financial management applications are using WinFS to store bills.

Different relations specify a different set of data. So when a search is made that encompasses multiple relations, the different sets of data are retrieved individually and a union of the different sets is computed. The resulting set contains only those data items that correspond to all the relations.

Notifications

WinFS also includes better support for handling data that changes frequently. Using WinFS Notifications, applications choose to be notified of changes to selected data Items. WinFS will raise an ItemChangedEvent, using the .NET Event model, when a subscribed-to Item changes, and the event will be published to the applications.

Data sharing

WinFS allows easy sharing of data between applications, and among multiple WinFS stores, which may reside on different computers, by copying to and from them. A WinFS item can also be copied to a non-WinFS file system, but unless that data item is put back into the WinFS store, it will not support the advanced services provided by WinFS.

The WinFS API also provides some support for sharing with non-WinFS applications. WinFS exposes a shell object to access WinFS stores. This object maps WinFS items to a virtual folder hierarchy, and can be accessed by any application. WinFS data can also be manually shared using network shares, by sharing the legacy shell object. Non-WinFS file formats can be stored in WinFS stores, using the File Item, provided by WinFS. Importers can be written, to convert specific file formats to WinFS Item types.

In addition, WinFS provides services to automatically synchronize items in two or more WinFS stores, subject to some predefined condition, such as "share only photos" or "share photos that have an associated contact X". The stores may be on different computers. Synchronization is done in a peer-to-peer fashion; there is no central authority. A synchronization can be either manual or automatic or scheduled. During synchronization, WinFS finds the new and modified Items, and updates accordingly. If two or more changes conflict, WinFS can either resort to automatic resolution based on predefined rules, or defer the synchronization for manual resolution. WinFS also updates the schemas, if required.

Application support

Shell namespace

WinFS Beta 1 includes a shell namespace extension, which surfaces WinFS stores as top level objects in My Computer view. Files can be copied into and out of the stores, as well as applications can be directly used to save there. Even folders such as My Documents can be redirected to the stores. WinFS uses Importer plug-ins to analyze the files as they were being imported to the store and create proper WinFS schemas and objects, and when taking the objects out, re-pack them into files. If importers for certain files are not installed, they are stored as generic File types.

Microsoft Rave

Microsoft Rave is an application that shipped with WinFS Beta 1. It allows synchronization of two or more WinFS stores, and supports synchronization in full mesh mode as well as the central hub topology. While synchronizing, Microsoft Rave will determine the changes made to each store since the last sync, and update accordingly. When applying the changes, it also detects if there is any conflict, i.e., the same data has been changed on both stores since the last synchronization. It will either log the conflicting data for later resolution or have it resolved immediately. Microsoft Rave uses peer-to-peer technology to communicate and transfer data.

StoreSpy

File:Storespy.jpg

WinFS included StoreSpy, a WinFS Item browser that could be used to browse structured data instances with their properties and relationships.

With WinFS Beta 1, Microsoft included an unsupported application called StoreSpy, which allowed one to browse WinFS stores by presenting a hierarchical view of WinFS Items. It automatically generated virtual folders based on access permissions, date and other metadata, and presented them in a hierarchical tree view, akin to what traditional folders are presented in. The application generated tabs for different Item types. StoreSpy allowed viewing Items, Relationships, MultiSet, Nested Elements, Extensions and other types in the store along with its full metadata. It also presented a search interface to perform manual searches, and save them as virtual folders. The application also presented a graphical view of WinFS Rules. However, it did not allow editing of Items or their properties, though it was slated for inclusion in a future release. But the WinFS project was cut back before it could materialize.

Type Browser

WinFS also includes another application, named WinFS Type Browser, which can be used to browse the WinFS types, as well as visualize the hierarchical relationship between WinFS types. A WinFS type, both built-in types as well as custom schemas, can be visualized along with all the properties and methods that it supports. It also shows the types that it derives from as well as other types that extend the type schema. However, while it was included with WinFS, it was released as an unsupported tool.

OPather

WinFS Beta 1 also includes an unsupported application, named OPather. It presents a graphical interface for writing Opath queries. It can be used by selecting target object type and specifying the parameters of the query. It also includes Intellisense-like parameter completion feature. It can then be used to perform visualization tasks like binding results of a query to a DataGrid control, create views of the data in WinFS itself, or just extract the query string.

Project "Orange"

Microsoft launched a project to build a data visualization application for WinFS. It was codenamed "Project Orange" and was supposedly built using Windows Presentation Foundation. It was supposed to provide exploration of Items stored in WinFS stores, and data relationships were supposed to be a prominent part of the navigation model. It was supposed to let people allow organization of the WinFS stores graphically as well – productizing many of the concepts shown in the IWish Concept Video WMV File. However, since the WinFS project went dark, the status of this project is unknown.

External links

• WinFS Blog
• WinFS at the Microsoft Developer Network (Internet Archive cache)
• Channel 9 Videos
• WinFS Newsgroup
• WinFS Beta 1 Preview
• WinFS area on NetFXGuide.com (Internet Archive cache)