User interaction in mixed reality interactive storytelling

Marc Cavazza, Olivier Martin, Fred Charles, Xavier Marichal, Steven Mead

Research output: Chapter in Book/Report/Conference proceedingConference contribution

236 Downloads (Pure)


One promising application of Mixed Reality to entertainment has been the development of “interactive theatre” systems involving synthetic actors. However, these systems do not make use of the most recent advances in interactive storytelling technologies, which use Artificial Intelligence techniques to generate real time narratives featuring synthetic characters and supporting user intervention in the plot. In this paper, the authors describe a Mixed Reality system based on a “magic mirror” model (Figure 1), in which the user’s image is captured in real time by a video camera, extracted from his/her background and mixed with a 3D graphic model of a virtual stage including the synthetic characters taking part in the story. The resulting image is projected on a large screen facing the user, who sees his/her own image embedded in the virtual stage with the synthetic actors. The graphic component of the Mixed Reality world is based on a game engine, Unreal Tournament 2003TM. This engine not only performs graphic rendering and character animation but incorporates a new version of our previously described storytelling engine [1]. A single 2D camera facing the user analyses the image in real-time by segmenting the user’s contours [2]. The objective behind segmentation is twofold. Firstly, it extracts the image silhouette of the user in order to be able to inject it into the virtual setting on the projection screen. Secondly, it can recognise user behaviour, including symbolic gestures, in real-time, hence supporting a new interaction channel.
Original languageEnglish
Title of host publicationProceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality
Place of PublicationWashington, DC
Number of pages2
ISBN (Print)0769520065
Publication statusPublished - 2003
Event2nd IEEE/ACM International Symposium on Mixed and Augmented Reality - Tokyo, Japan
Duration: 7 Oct 200310 Oct 2003
Conference number: 2nd


Conference2nd IEEE/ACM International Symposium on Mixed and Augmented Reality
Abbreviated titleISMAR '03

Bibliographical note

Author can archive publisher's version/PDF.


Dive into the research topics of 'User interaction in mixed reality interactive storytelling'. Together they form a unique fingerprint.

Cite this