In this chapter, we discuss mixed-media access, an information access paradigm for multimedia data in which the media type of a query may differ from that of the data. This allows a single query to be used to retrieve information from data consisting of multiple types of media. In addition, multiple queries formulated in different media types can be used to more accurately specify the data to be retrieved. The types of media considered in this paper are speech, images of text, and full-length text. Some examples of metadata for mixed-media access are locations of keywords in speech and images, identification of speakers, locations of emphasized regions in speech, and locations of topic boundaries in text. Algorithms for automatically generating this metadata are described, including word spotting, speaker segmentation, emphatic speech detection, and subtopic boundary location. We illustrate the use of mixed-media access with an example of information access from multimedia data surrounding a formal presentation.

The discipline of narratology has long recognized the need to classify documents as instances of different text types. We have discovered that classification is as applicable to hypermedia as it is to any other document presentation. Following the work of S. Chatman (1978; 1990), we consider three such text types: description, argument and narrative. The goal of a description document is to describe some object or concept; this is usually achieved by describing component parts and then describing how those parts combine to constitute the entirety. An argument document, on the other hand, is concerned with establishing some assertion or point of view, and it is based on supporting evidence, as well as possible refutations and justifications for defeating those regulations. Finally, a narrative document recounts some sequence of events in time, addressing relationships such as causality and contingency among those events. We analyze these types through case studies that give an example of each as a hypermedia document. We then argue that this classification provides an organizational framework that facilitates the construction of outlines that serve the writer in preparing the actual content of a document. Such outlines can also benefit the reader's understanding of the content that the writer intended to convey; if the writer does not make those outlines available explicitly to the reader, the reader can use knowledge of the document type to construct his own version of those outlines. Finally, we review some early work in content based indexing and search of multimedia documents

S.W. Smoliar, J.D. Baker, T. Nakayama, and L. Wilcox,

An approach to semantics based on traditional paradigms of knowledge representation (e.g., developing reductionist models of video document genres), while not entirely of the mark, may be significantly misdirected. Understanding the semantics of video and multimedia must begin with understanding how video (and film) are "read" and "written." The purpose of this paper is to set an agenda for coming to an understanding of reading and writing multimedia and to address the representationalist implications of achieving that end. Further, we illustrate this research agenda, showing how we have applied concepts and methods from film theory (to the reading process) and interaction analysis (to the work of multimedia production), and what our preliminary findings might mean for computational support and knowledge representation in multimedia.

The fundamental objective of Extended Media research is the empowering of documents through technology. We see our goal as that of inventing documents which communicate more effectively. Furthermore, we need to make it easier for the writer to record what must be communicated and for the reader to access it. Thus, we anticipate that one cannot invent new documents without also inventing new processes for both writing and reading. Our primary research thrust is thus in the authoring of hypermedia documents, supplemented by a secondary thrust concerned with the problem of managing archives and libraries where more than text is involved.

We have developed a technique that categorizes document images based on their content. Unlike conventional methods that use optical character recognition (OCR), we convert document images into word shape takens, a shape-based representation of words. Because we have only to recognize simple graphical features from image, this process is much faster than OCR. Although the mapping between word shape tokens and words is one-to-many, they are a rich source of information for content characterization. Using a vector space classifier with a scanned document image database, we show that the word shape token-based approach is quite adequate for content-oriented categorization in terms of accuracy compared with conventional OCR-based approaches.

Online digital audio is a rapidly growing resource, which can be accessed in rich new ways not previously possible. For example, it is possible to listen to just those portions of a long discussion which involve a given subset of people, or to instantly skip ahead to the next speaker. Providing this capability to users, however, requires generation of necessary indices, as well as an interface which utilizes these indices to aid navigation. We describe algorithms which generate indices from automatic acoustic segmentation. These algorithms use hidden Markov models to segment audio into segments corresponding to different speakers or acoustics classes (e.g. music). Unsupervised model initialization using agglomerative clustering is described, and shown to work as well in most cases as supervised initialization. We also describe a user interface which displays the segmentation in the form of a timeline, which tracks for the different acoustic classes. The interface can be used for direct navigation through the audio.

Trends such as the massive increase in information available via electronic networks, the use of on-line product data by distributed concurrent engineering teams, and dynamic supply chain integration for electronic commerce are placing severe burdens on traditional methods of information sharing and retrieval. Sources of information are far too numerous and dynamic to be found via traditional information retrieval methods, and potential consumers are seeing increased need for automatic notification services. Matchmaking is an approach based on emerging information integration technologies whereby potential producers and consumers of information send messages describing their information capabilities and needs. These descriptions, represented in rich, machine-interpretable description languages, are unified by the matchmaker to identify potential matches. Based on the matches, a variety of information brokering services are performed. We introduce matchmaking, and argue that it permits large numbers of dynamic consumers and providers, operating on rapidly-changing data, to share information more effectively than via traditional methods. Two matchmakers are described, the SHADE matchmaker, which operates over logical and structured text languages, and the COINS matchmaker, which operates over free text. These matchmakers have been used for a variety of applications, most significantly, in the domains of engineering and electronic commerce. We describe our experiences with the SHADE and COINS matchmaker, and we outline the major observed benefits and problems of matchmaking.

As agents see more use in dynamic, distributed information networks, information sharing facilitators, such as the SHADE matchmaker, and underlying knowledge-based agent communication protocols, such as the Knowledge Query and Manipulation Language, will see increased use. We have created several communities of agents collaborating via KQML and matchmaking within the domains of collaborative engineering and satellite image retrieval. Based on these experiences, matchmaking has proven to be very beneficial for multi-agent systems, but we have also identified a number of issues and extensions that are not only vital to KQML-based matchmaking, but to inter-agent protocols in general. These include representational approaches to advertising complex databases, approaches to error recovery and response timing, maintaining consistency among information providers, scalability, security, persistent requests in information brokering, and the dilemma between explicit vs. implicit brokering.

Structuring temporal relationships among multimedia information elements is one of the most important facilities for editing and creating multimedia documents. We have developed a multimedia system named MediaPreview which provides facilities for structuring and creating multimedia documents. In this paper, we present a document model which is adopted in MediaPreview. This model has been designed to realize spatial and temporal structuring for the documents. The main feature of this model is the concept of a "Multimedia Paragraph" (MMP) which is introduced to reduce the complexity of the temporal structuring, such as the asynchronous interactive operation among documents. The concept of "MMP" provides an explicit and basic unit which is used to create a document in a "top-down" manner. This paper also presents the system architecture and implementation of MediaPreview in a distributed environment including database system facilities. This system realizes "static" and "dynamic" integration schemes for multimedia information elements. Our system includes a parallel database engine which manipulates multimedia information elements as streams. This database engine is effectively used for creating multimedia documents.