Digestor is a software system which automatically re-authors arbitrary documents from the world-wide web to display appropriately on small screen devices such as PDAs and cellular phones, providing device-independent access to the web. Digestor is implemented as an HTTP proxy which dynamically re-authors requested web pages using a heuristic planning algorithm and a set of structural page transformations to achieve the best looking document for a given display size.

Traditionally hypertexts have been limited in size by the manual effort required to create hypertext links. In addition, large hyper-linked collections may overwhelm users with the range of possible links from any node, only a fraction of which may be appropriate for a given user at any time. This work explores automatic methods of link construction based on feedback from users collected during browsing. A full-text search engine mediates the linking process. Query terms that distinguish well among documents in the database become candidate anchors; links are mediated by passage-based relevance feedback queries. The newspaper metaphor is used to organize the retrieval results. VOIR, a software prototype that implements these algorithms has been used to browse a 74,500 node (250MB) database of newspaper articles. An experiment has been conducted to test the relative effectiveness of dynamic links and user-specified queries. Experimental results suggest that link-mediated queries are more effective than user-specified queries in retrieving relevant information. The paper concludes with a discussion of possible extensions to the linking algorithms.

Dynomite is a portable electronic notebook for the capture and retrieval of handwritten and audio notes. The goal of Dynomite is to merge the organization, search, and data acquisition capabilities of a computer with the benefits of a paper-based notebook. Dynomite provides novel solutions in four key problem areas. First, Dynomite uses a casual, low cognitive overhead interface. Second, for content indexing of notes, Dynomite uses ink properties and keywords. Third, to assist organization, Dynomite's properties and keywords define views, presenting a subset of the notebook content that dynamically changes as users add new information. Finally, to augment handwritten notes with audio on devices with limited storage, Dynomite continuously records audio, but only permanently stores those parts highlighted by the user.

Video-based awareness tools increase familiarity among remote group members and provide pre-communication information. Low-cost iconic indicators provide less but more succinct information than video images while preserving privacy. Observations of and feedback from users of our video awareness tool suggest that an activity sensing feature along with a variety of privacy options combines advantages of both the video images and iconic indicator approaches. We introduced the activity sensing feature in response to user requests. It derives activity information from video images and provides options to control privacy and improves the usability of video-based awareness tools.

Dynomite is a portable electronic notebook that merges the benefits of paper note-taking with the organizational capabilities of computers. Dynomite incorporates four complementary features which combine to form an easy to use system for the capture and retrieval of handwritten and audio notes. First, Dynomite has a paper-like user interface. Second, Dynomite uses ink properties and keywords for content indexing of notes. Third, Dynomite's properties and keywords allows retrieval of specific ink and notes. The user is shown a view, or a subset of the notebook content, that dynamically changes as new information is added. Finally, Dynomite continuously records audio, but only permanently stores highlighted portions so that it is possible to augment handwritten notes with audio on devices with limited storage.

Hypertext interfaces are considered appropriate for information exploration tasks. The prohibitively expensive link creation effort, however, prevents traditional hypertext interfaces from being used with large coherent collections of text. Such collections typically require query-based interfaces. This paper examines a hybrid approach: the system described here creates anchors dynamically based on users' queries, and uses anchor selection as a query expansion mechanism. An experiment was conducted to compare browsing behavior in query- and link-based interfaces. Results suggest that query-mediated links are as effective as explicit queries, and that strategies adopted by users affect performance. This work has implications for the design of information exploration interfaces; the dynamic link algorithms described here are being incorporated into a Web server.

Palplates are a collection of touch-screen terminals placed around the office enabling human-computer interactions at the point of need. Supporting a community of mobile authenticated workers with a small number of stationary devices is an alternative to providing each person with a portable wireless computer. In contrast to the PC's desktop metaphor, Palplates use a place metaphor that reflect the actual rooms, corridors, and buildings that are part of the office place. Users interact graphically with applications supported by a geographic database. The user interface is generated dynamically based on the user's identity, the point-of-access, and the changing collection of physical office equipment, electronic documents and applications present at any given location.

In this chapter, we discuss mixed-media access, an information access paradigm for multimedia data in which the media type of a query may differ from that of the data. This allows a single query to be used to retrieve information from data consisting of multiple types of media. In addition, multiple queries formulated in different media types can be used to more accurately specify the data to be retrieved. The types of media considered in this paper are speech, images of text, and full-length text. Some examples of metadata for mixed-media access are locations of keywords in speech and images, identification of speakers, locations of emphasized regions in speech, and locations of topic boundaries in text. Algorithms for automatically generating this metadata are described, including word spotting, speaker segmentation, emphatic speech detection, and subtopic boundary location. We illustrate the use of mixed-media access with an example of information access from multimedia data surrounding a formal presentation.

The discipline of narratology has long recognized the need to classify documents as instances of different text types. We have discovered that classification is as applicable to hypermedia as it is to any other document presentation. Following the work of S. Chatman (1978; 1990), we consider three such text types: description, argument and narrative. The goal of a description document is to describe some object or concept; this is usually achieved by describing component parts and then describing how those parts combine to constitute the entirety. An argument document, on the other hand, is concerned with establishing some assertion or point of view, and it is based on supporting evidence, as well as possible refutations and justifications for defeating those regulations. Finally, a narrative document recounts some sequence of events in time, addressing relationships such as causality and contingency among those events. We analyze these types through case studies that give an example of each as a hypermedia document. We then argue that this classification provides an organizational framework that facilitates the construction of outlines that serve the writer in preparing the actual content of a document. Such outlines can also benefit the reader's understanding of the content that the writer intended to convey; if the writer does not make those outlines available explicitly to the reader, the reader can use knowledge of the document type to construct his own version of those outlines. Finally, we review some early work in content based indexing and search of multimedia documents

S.W. Smoliar, J.D. Baker, T. Nakayama, and L. Wilcox,

An approach to semantics based on traditional paradigms of knowledge representation (e.g., developing reductionist models of video document genres), while not entirely of the mark, may be significantly misdirected. Understanding the semantics of video and multimedia must begin with understanding how video (and film) are "read" and "written." The purpose of this paper is to set an agenda for coming to an understanding of reading and writing multimedia and to address the representationalist implications of achieving that end. Further, we illustrate this research agenda, showing how we have applied concepts and methods from film theory (to the reading process) and interaction analysis (to the work of multimedia production), and what our preliminary findings might mean for computational support and knowledge representation in multimedia.

The fundamental objective of Extended Media research is the empowering of documents through technology. We see our goal as that of inventing documents which communicate more effectively. Furthermore, we need to make it easier for the writer to record what must be communicated and for the reader to access it. Thus, we anticipate that one cannot invent new documents without also inventing new processes for both writing and reading. Our primary research thrust is thus in the authoring of hypermedia documents, supplemented by a secondary thrust concerned with the problem of managing archives and libraries where more than text is involved.

We have developed a technique that categorizes document images based on their content. Unlike conventional methods that use optical character recognition (OCR), we convert document images into word shape takens, a shape-based representation of words. Because we have only to recognize simple graphical features from image, this process is much faster than OCR. Although the mapping between word shape tokens and words is one-to-many, they are a rich source of information for content characterization. Using a vector space classifier with a scanned document image database, we show that the word shape token-based approach is quite adequate for content-oriented categorization in terms of accuracy compared with conventional OCR-based approaches.

Online digital audio is a rapidly growing resource, which can be accessed in rich new ways not previously possible. For example, it is possible to listen to just those portions of a long discussion which involve a given subset of people, or to instantly skip ahead to the next speaker. Providing this capability to users, however, requires generation of necessary indices, as well as an interface which utilizes these indices to aid navigation. We describe algorithms which generate indices from automatic acoustic segmentation. These algorithms use hidden Markov models to segment audio into segments corresponding to different speakers or acoustics classes (e.g. music). Unsupervised model initialization using agglomerative clustering is described, and shown to work as well in most cases as supervised initialization. We also describe a user interface which displays the segmentation in the form of a timeline, which tracks for the different acoustic classes. The interface can be used for direct navigation through the audio.

Trends such as the massive increase in information available via electronic networks, the use of on-line product data by distributed concurrent engineering teams, and dynamic supply chain integration for electronic commerce are placing severe burdens on traditional methods of information sharing and retrieval. Sources of information are far too numerous and dynamic to be found via traditional information retrieval methods, and potential consumers are seeing increased need for automatic notification services. Matchmaking is an approach based on emerging information integration technologies whereby potential producers and consumers of information send messages describing their information capabilities and needs. These descriptions, represented in rich, machine-interpretable description languages, are unified by the matchmaker to identify potential matches. Based on the matches, a variety of information brokering services are performed. We introduce matchmaking, and argue that it permits large numbers of dynamic consumers and providers, operating on rapidly-changing data, to share information more effectively than via traditional methods. Two matchmakers are described, the SHADE matchmaker, which operates over logical and structured text languages, and the COINS matchmaker, which operates over free text. These matchmakers have been used for a variety of applications, most significantly, in the domains of engineering and electronic commerce. We describe our experiences with the SHADE and COINS matchmaker, and we outline the major observed benefits and problems of matchmaking.

As agents see more use in dynamic, distributed information networks, information sharing facilitators, such as the SHADE matchmaker, and underlying knowledge-based agent communication protocols, such as the Knowledge Query and Manipulation Language, will see increased use. We have created several communities of agents collaborating via KQML and matchmaking within the domains of collaborative engineering and satellite image retrieval. Based on these experiences, matchmaking has proven to be very beneficial for multi-agent systems, but we have also identified a number of issues and extensions that are not only vital to KQML-based matchmaking, but to inter-agent protocols in general. These include representational approaches to advertising complex databases, approaches to error recovery and response timing, maintaining consistency among information providers, scalability, security, persistent requests in information brokering, and the dilemma between explicit vs. implicit brokering.

Structuring temporal relationships among multimedia information elements is one of the most important facilities for editing and creating multimedia documents. We have developed a multimedia system named MediaPreview which provides facilities for structuring and creating multimedia documents. In this paper, we present a document model which is adopted in MediaPreview. This model has been designed to realize spatial and temporal structuring for the documents. The main feature of this model is the concept of a "Multimedia Paragraph" (MMP) which is introduced to reduce the complexity of the temporal structuring, such as the asynchronous interactive operation among documents. The concept of "MMP" provides an explicit and basic unit which is used to create a document in a "top-down" manner. This paper also presents the system architecture and implementation of MediaPreview in a distributed environment including database system facilities. This system realizes "static" and "dynamic" integration schemes for multimedia information elements. Our system includes a parallel database engine which manipulates multimedia information elements as streams. This database engine is effectively used for creating multimedia documents.