Figure 1: Disparate Web Services independently publishing data about a common set of objects
TAP is a system for knitting together data fragments from disparate XML/SOAP based Web Services into a single schematically unified global knowledge base. TAP provides a set of guidelines for the publication of these data fragments. These guidelines coordinate the on-demand assembly of the fragments into a coherent knowledge base. A lightweight TAP library provides a minimalist API using which an application can pull in and integrate fragments of the distributed knowledge base as required. A number of technical issues arise in building such a system. Foremost is the problem of reference, i.e., how does a client refer to an object when requesting a server for some data about that object, without requiring the server and client to pre-agree on names for every object they might want to exchange data about? We propose Reference by Descriptions as a solution to this important problem. We also describe Semantic Search, an application of this global knowledge base to the problem of searching for data across multiple dynamically changing web sites.
A number of these Web Services will be transactional, e.g., for making reservations, buying products, etc. We also expect a significant number of web sites to provide SOAP interfaces that simply make the information currently published in html also available in XML. For example, today, there is a lot of information on the web about music and musicians. Given any particular musician, such as Yo-Yo Ma, there are a number of websites with different pieces of information about him --- TicketMaster has his concert schedule, Amazon and CDNow have his discography, AllMusic has biographical information … In addition there are numerous fan sites with assorted tidbits of information about him.
Many of these sites already make their data available to their partners. The data is made available as XML files which their partners access in batch mode. A number of sites (such as Yahoo, AOL and MSN) collect data (about a particular topic such as music or travel) from multiple partners, aggregate them and provide them to their users. The task of aggregation is expensive and often laborious. Some of the factors that complicate this aggregation include the lack of standardized schemas, the use of different names for the same entity by different sites and the latency associated with batch processes that makes the data stale.
The emergence of standards (and hence tools, etc.) for Web Services will enable sites to make their data available in an interactive fashion, i.e., the consumer of the data can access just the relevant data, as it needs it by issuing SOAP queries. Moving from a batch mode to an interactive mode also up new avenues for simplifying the aggregation process. TAP hopes to use this shift to simply the aggregation to a point where that the data coming out of all these Web Services is effectively one global knowledge base.
TAP is an implemented system, and a continuing project. We expect to make the source code and knowledge bases associated with TAP freely available on the web, sometime during the summer of 2002.
The goal is that a small program can take these independently produced pieces of data and integrate them into a coherent whole as illustrated in figure 2.
TAP provides a set of guidelines for the publication of these data fragments. These guidelines coordinate the on-demand assembly of the fragments into the whole. A simple, lightweight TAP library provides a minimalist API using which an application can pull in and integrate fragments of the distributed knowledge base as required. Section 3 describes the architecture of the system. Section 4 lists some of the key technical problems and their solutions .
The widespread adoption of a system like TAP, and the Semantic Web [3] in general is going to be driven by applications that provide a compelling reason for sites to make their data available according to TAP guidelines. For this reason, we are also building a couple of key applications for TAP. Section 5.1 describes an application we are building, called Semantic Search, which uses TAP based results complement traditional search results.
This section describes the TAP architecture. The core architectural principles underlying TAP are:
XML provides an adequate syntax layer for encoding data.
However, XML by itself does not provide a mechanism for
aggregation. There is no domain independent way of
aggregating two arbitrary chunks of XML. Further, XML does not have the notion
of object identity. Given two chunks of XML, both of which refer to a
real-world object (such as Yo-Yo Ma), there is no simple way of resolving both
of these references to the same object. In order to enable easy aggregation of
the XML published by disparate Web Services, we need to use a data model that
the XML published by these Web Services conforms to.
TAP currently uses directed labeled graphs (i.e., RDF [4] without containers or reification) as its data model (in the remainder of this paper, when we say "RDF", we mean RDF without containers or reification). Each site exposes its data as a directed labeled graph. The graph provided by a single site is referred to as its local graph and the knowledge base that emerges from the aggregation of these local graphs is referred to as the global graph. Graph super-positioning aggregates the global graph from the local graphs. The tricky part in this process is the proper alignment of nodes and arcs from different servers. Nodes that correspond to the same object, from different servers, need to be to unified/merged in the aggregation process. The co-identification of nodes from different sources is intimately tied to way nodes are referenced. This is covered in section 4.1.
We are also exploring more complex, expressive and heterogeneous models. Any model that supports aggregation and object identity should work. One approach being explored is to extend XML schemas [5] so that they can specify, at a domain specific level, how to perform aggregation and object identity resolution.
At the core of TAP is an interface called GetData. GetData is the SOAP interface by which sites publish their local graph. It is hence also the interface using which an application navigates the global graph. The GetData interface, as implemented by a server, takes two arguments: (1) Reference to a node and (2) the name/URI of an arc label/property type It is expected to return the value(s) of that property type for that node. It is allowed (and encouraged) for the server to return more information about each of the values. When the server has an answer for the GetData request, it responds with a SOAP message whose body has a Data element that contains the answer. In other cases (such as when there is an ambiguity in reference), the server responds with other messages.
On the client, GetData takes an additional argument, the server to which the SOAP query is addressed. GetData is the interface with which a client program incrementally explores the global graph. We also have an interface called GetDataInverse, which allows for the reverse traversal of arcs. Here are a couple of examples of GetData.
GetData(<Reference to Yo-Yo Ma>, birthplace) = <Reference to Paris, France>
GetData(<Reference to Paris, France>, Temperature) => 59 degrees F
Appendix A lists some sample exchanges using GetData.
We expect the emergence of servers that know of reliable sources of different kinds of data about different categories of objects. These servers can also act as aggregation points, so that clients can send their queries to these servers which redirect them. Such servers, which can also cache the data, can also provide more complex and sophisticated query interfaces built on top of GetData. We are currently building such a server as part of the TAP project.
TAP in general and GetData in particular have been inspired by DNS. DNS provides a unified interface to a database that is distributed all over the Internet. GetHostByName, the core DNS interface, takes a string denoting a host and returns its IP address. GetData is a generalization of GetHostByName. Instead of just retrieving a particular property (IP Address) of one kind of object (Internet hosts), GetData aims to work for a large number of different properties for a large number of different kinds of objects.
There are a number of technical issues that arise in the implementation of GetData. We briefly describe some these issues and go into some depth on one of them, the problem of Reference. There are other systems level issues that will be discussed elsewhere.
In the above examples of GetData, we need a way of referring to nodes like <Paris, France> and <Yo-Yo Ma> such that an arbitrary client can easily generate them and an arbitrary server can interpret, without prior agreement on specific names for each of the objects that they might want to exchange data about. The number of property types (like temperature and birthplace) is relatively small, at least compared to the number of nodes, which is potentially in the many billions. RDF, like most Web related formats, relies on URIs to identify and refer to nodes. URIs should be adequate to refer to property types. However, given the number of nodes, we believe that URIs as the sole mechanism for referring to nodes is not scalable from the perspective of coordination complexity. Getting everyone to agree on a set of names/URIs/URNs for every object they might want to exchange data about is not realistic.
Given any object (such as the album "Appalachian Journey" in figure 1), there will be number of different sites with data about that object. In reality, we do not get a nice set of local graphs as shown in figure 1, but instead get something that looks more like figure 3.
A program assembling the global graph from the fragments with disperse origins has to resolve references (such as 1136120489 and Bafixql5ld6e) from different sites to the same object (Yo-Yo Ma). Straight forward approaches such as assigning every object a unique canonical name (a URI) involve getting millions of sites to agree on names for billions of objects leading to potentially insurmountable coordination complexities.
This problem is different from that of URLs, where every page is controlled by a single site and the creator of the object gives it a name when the object is created. In contrast, no one controls Beethoven (or Yo-Yo Ma) or any number of other objects. More importantly, a large number of sites have data about Beethoven and Yo-Yo Ma. Even if someone were to be designated as the URN assigner for Yo-Yo Ma (such as his publisher or even him), we would still have a problem mapping the identifiers/keys currently in use by different databases to this URI.
For example, Amazon.com's internal name for the album Appalachian Journey is B00004Sh38H and CDNow's internal name is 757707. Data from these two sources about Yo-Yo Ma's discography will have to refer to Appalachian Journey. If each of them uses their internal names, a program aggregating data from CDNow and Amazon.com will have to cross-map these different names. Alternately, both of them can agree to use the URI assigned by the URI assigner for People/Musicians/Composers (we already see a problem here!), which leads to the problem of getting all these publishers each to engage in a giant mapping project. Further, as new albums are released and new musicians appear, these sites have to coordinate with the URI assignment process for these new objects. Both these approaches don't scale because of the coordination complexities involved.
We introduce Reference by Description as a way around this coordination nightmare. A description enables one machine to identify an object to another machine without a prior agreement on a name for that object. Instead of agreeing on names for objects, they agree on a way of describing objects (i.e., standard description language and vocabulary). The client and server use descriptions to refer to objects in GetData calls and in the returned values. Descriptions introduce a level of indirection that should significantly reduce the number of terms that all the participants need to agree on. Names, both local and global are just degenerate descriptions.
In the above example, each of the sources has to interpret and dereference a description of Appalachian Journey, which in this case could be (an encoding in a formal language of) "a MusicAlbum whose title is 'Appalachian Journey' which hasAuthor a Musician whose title is 'Yo-Yo Ma'". The terms in italics come from a standard vocabulary shared by the client and server. The core assumption is that by using a relatively small core vocabulary and standard system for description, servers and clients can bootstrap to a mutually comprehensible scheme for referring to a very large (compared to the size of the core vocabulary) set of objects.
We have created a core vocabulary of property types and classes for creating descriptions of large number of popular consumer objects related to music, sports, places, etc. These terms are arranged in taxonomy, with each of the property types assigned domain/range constraints as per RDF Schema [7]. We plan to enhance it in the near future to use DAML+OIL [8] In its current state, this vocabulary is adequate to provide descriptions of most of the items on sale at the top ecommerce sites such as Amazon and AOL shopping.
The standard vocabulary need not all come from one source. We provide a core kernel of terms, which can be extended by communities and their applications. Some of the terms defined by the applications will over a period of time get absorbed into the kernel. The use of namespace qualifications allows data providers to define their own extensions. The evolution of this ecology of names will likely mirror the evolution of operating system APIs, wherein, over a period of time, the operating systems incorporated the APIs offered by the more successful platform-like applications running on the operating systems.
Formally, a description is an RDF fragment corresponding to either a <em>description</em> or <em>typed Node</em> of the grammar specified in the RDF Model and Syntax specification. The description of Appalachian Journey given earlier would be encoded as (namespace declarations have been dropped to enhance readability) (Appendix A has many more examples of this):
<a:MusicAlbum>
<dc:title>Appalachian Journey</dc:title>
<a:hasAuthor>
<a:Musician>
<dc:title>Yo-Yo Ma</dc:title>
</a:Musician>
</a:hasAuthor>
</a:MusicAlbum>
For the description to work, the server and client will need to share the meaning of the non-literals in the RDF --- MusicAlbum, title, Musician and hasAuthor. The benefit is that sharing this small set of terms (plus a few more like MusicGroup), enables the parties involved to bootstrap to interoperable references for all musicians and music albums.
A description of an object is intended to uniquely refer and identify the object in the context of the recipient. Ideally, there should be only one object matching the description in the domain of objects that the server (which is being queried) knows about. Objects might not, and in fact, typically do not have canonical descriptions. Depending on who is issuing the query and who is receiving the query, different descriptions, more or less detailed, may be used.
Reference by Descriptions is a form of loose coupling. Whenever there is loose coupling, there is the possibility of a failure to couple. In the case of Reference by Descriptions, there are a few different failure cases, which need to be handled. The most important of these is when there is an ambiguity. The description might refer to more than one object. For example, a client might want to find out some piece of information about Pink Floyd's Dark Side of the Moon Album and might use the description "A MusicAlbum whose title is 'Dark Side of the Moon' which hasAuthor a MusicGroup whose title is 'Pink Floyd'". As it turns out, there are two distinct albums that match this description --- the original release and the twenty fifth-anniversary special edition. Assuming the server is aware of both these albums, it responds with a Choose message, providing descriptions of the alternative denotations of the description. The client can pick a denotation and reissue the query. Future extensions to GetData should enable the client to tell the server, as part of a GetData request, to pick any denotation in the case of ambiguity.
There are other cases where the server might have the object being referenced, but is not able to interpret the description as referring to that object. For example, in the previous example, an overzealous client may use the description "a MusicAlbum whose title is 'Appalachian Journey' which hasAuthor a Musician whose title is 'Yo-Yo Ma' and who was born in a City whose title is Paris". The server, not knowing that Yo-Yo Ma was born in Paris might not be able to map the description to the object. On the other hand, if there are multiple Yo-Yo Mas, then, this piece of information might be useful in the disambiguation. In this case, it should be possible for the server and client to enter into a negotiation in which they can progressively weaken/strengthen the description until both are confident that they are referring to the same object.
Descriptions work when the client (A) and the server (B) share some knowledge about the object they are exchanging data about and when this shared knowledge is enough to distinguish the object from other objects in the domain of the interpreter of the description. If this shared knowledge is too small, the communication might not be possible. In this case it is quite possible that a third party (C) could be brought in to help with the generation and interpretation of descriptions. The third party should know about this object and its knowledge about this object should overlap that of both the parties, such that both A intersection C and B intersection C are adequate to identify the object to C. In the longer run, we would like to see a general theory of references, which covers these and other issues.
Applications are the key to driving the creation and adoption of TAP. Different kinds of data will have different applications. In this section, we describe one interesting application of TAP, Semantic Search, which applies TAP to the problem of searching for information on the Internet.
Semantic Search (SS) is an application of TAP to search. TAP based results complement traditional search results. Given the search query, Semantic Search retrieves real-time data relevant to that query from TAP. Internet search engines are primarily targeted at text and not data. Further, the crawl, grab, and index model of search does not work well for searching across dynamically changing sites. Semantic Search uses TAP to overcome these two shortcomings of search.
Semantic Search also introduces the concept of Activity Based Search. Users search (the Internet, a particular site, ...) in the context of an activity. Even a very shallow understanding of the potential activities that make sense in the context of a given search can make a dramatic difference in the search experience. We generalize what Google does when the user types in an address (like 2231 Fallen Leaf Ave, Los Altos, CA) into the search box. It identifies that the search term is an address and provides a link to Mapquest for that address, because getting a map is one of the common activities associated with street addresses.
Given the search term, we look up the term in the Semantic Search Knowledge Base (described below). If the term is found in the Knowledge Base (KB), based on the type of the concept it denotes, we determine the kinds of activities that are typically associated with that concept. Based on that, we determine the kinds of data (i.e., property types of the concept) from the global graph that should be used to augment the search results. This data is fetched from the global graph using GetData and used to augment traditional search results. In the case that the search term has multiple denotations (e.g., "jaguar" could denote the animal or the car), the system selects one and offers the user the ability to choose the other denotation.
Example: Given the search term "Yo-Yo Ma", we figure out that Yo-Yo Ma is a musician. The activities associated with a musician are (retrieving data about) his concert schedule, albums, posters, auctions, ... The system uses TAP to retrieve these pieces of data, which are then used to augment the search results. In this example, the search results are augmented with the data that Yo-Yo Ma is performing in Seattle on 5/12 (from TicketMaster), that there is an auction for one of his CDs on EBay which ends in 3 minutes, that CDNow has his Appalachian Journey CD on sale for $14.99, ...
When applied to ecommerce data (as above), Semantic Search creates a new kind of advertising, i.e., real-time advertising. This provides a business reason for search engines and ecommerce vendors to adopt Semantic Search. In order to prime the pump, we have written scrapers for about 50 of the top ecommerce sites to provide data for SS. Figure 4 shows a screen-shot of a search for Yo-Yo Ma, augmented with Semantic Search.
The global graph is about objects such as people, places and events and not about text strings. In order to retrieve data from the graph, Semantic Search first needs to identify the concept(s) that the search term denotes. To do this it uses a large knowledge base we have constructed.
The knowledge base is built on top of the ontology of core terms that is used for descriptions. To this core ontology, it adds basic information about particular objects, such as particular musicians, particular athletes and particular places. As of February 2002, the Knowledge Base has data about hundred thousand nodes. Here are some categories of objects that the KB currently covers.
Different parts of the knowledge base are built and maintained using different tools. We have built a simple web based tool for simple manual editing of the knowledge base. Parts of the knowledge base were obtained by mining free data sources such as the Open Directory Project. Some portions of the knowledge base are grown and maintained in an automated fashion by a system that reads news articles and identifies new musicians, movies, actors, etc.
The contents of this knowledge base are themselves available as part of the global graph. We hope that this will help bootstrap the graph. This knowledge base is intended to complement, not replace, systems like Cyc [6], which have a deep knowledge about basic, common sense phenomenon, but don’t have knowledge about particulars. So, for example, Cyc knows a lot about what it means to be a Musician. If it is told that Yo-Yo Ma is a Cellist, it can infer that he probably owns one or more Cellos, plays the Cello often, etc. But it might know that there is a famous Cellist called Yo-Yo Ma (it certainly does know all the famous composers, classical instrumentalists and opera singers!) The Semantic Search KB is also not intended to replace the so-called “Upper Ontologies”. It should be possible to hoist the Semantic Search KB on top of any of those other core ontologies.
Request message:
<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/1999/XMLSchema"
xmlns:a="http://www.alpiri.org/">
<SOAP-ENV:Body>
<GetData>
<a:Musician>
<a:title>Yo Yo Ma</a:title>
<a:title>Ma, Yo-Yo</a:title>
<a:playsInstrument resource="http://www.alpiri.org/CelloInstrument"/>
<a:oid namespace="a">MusicianMa,_Yo-Yo</a:oid>
</a:Musician>
<a:concertSchedule/>
</GetData>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
End of request.
Response message:
<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/1999/XMLSchema"
xmlns:a="http://www.alpiri.org/">
<SOAP-ENV:Body>
<Data>
<a:Concert>
<a:title>Silk Road Project With Yo-Yo Ma</a:title>
<a:oid namespace="ticketmaster">F003384F15CA4E4</a:oid>
<a:date>5/12/02</a:date>
<a:time>7:30PM</a:time>
<a:eventLocation>
<a:Auditorium>
<a:title>Benaroya Hall</a:title>
<a:location>
<a:City>
<a:title>Seattle</a:title>
<a:location>
<a:USState>
<a:title>Washington</a:title>
</a:USState>
</a:location>
</a:City>
</a:location>
</a:Auditorium>
</a:eventLocation>
</a:Concert>
<a:Concert>
<a:title>Silk Road Project with Yo-Yo Ma</a:title>
<a:oid namespace="ticketmaster">F003384F7EDB26E</a:oid>
<a:date>5/13/02</a:date>
<a:time>7:30PM</a:time>
<a:eventLocation>
<a:Auditorium>
<a:title>Benaroya Hall</a:title>
<a:location>
<a:City>
<a:title>Seattle</a:title>
<a:location>
<a:USState>
<a:title>Washington</a:title>
</a:USState>
</a:location>
</a:City>
</a:location>
</a:Auditorium>
</a:eventLocation>
</a:Concert>
</Data>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
End of response.
Request message:
?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/1999/XMLSchema"
xmlns:a="http://www.alpiri.org/">
<SOAP-ENV:Body>
<GetData>
<a:Musician>
<a:title>Yo Yo Ma</a:title>
<a:title>Ma, Yo-Yo</a:title>
<a:playsInstrument resource="http://www.alpiri.org/CelloInstrument">
<a:oid namespace="a">MusicianMa,_Yo-Yo</a:oid>
</a:Musician>
<a:hasAuthor inverse="yes">
</GetData>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
End of request.
Response message:
?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/1999/XMLSchema"
xmlns:a="http://www.alpiri.org/">
<SOAP-ENV:Body>
<Data>
<a:MusicAlbum>
<a:title>Appalachian Journey</a:title>
<a:hasAuthor>
<a:Musician>
<a:title>Yo-Yo Ma</a:title>
</a:Musician>
</a:hasAuthor>
<a:oid namespace="cdnow">757707</a:oid>
<a:price>$10.99</a:price>
</a:MusicAlbum>
<a:MusicAlbum>
<a:title>Tavener: The Protecting Veil</a:title>
<a:hasAuthor>
<a:Musician>
<a:title>Yo-Yo Ma</a:title>
</a:Musician>
</a:hasAuthor>
<a:oid namespace="cdnow">559275</a:oid>
<a:price>$17.99</a:price>
</a:MusicAlbum>
</Data>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
End of response.