User interaction in mixed reality interactive storytelling

One promising application of Mixed Reality to entertainment has been the development of “interactive theatre” systems involving synthetic actors. However, these systems do not make use of the most recent advances in interactive storytelling technologies, which use Artificial Intelligence techniques to generate real time narratives featuring synthetic characters and supporting user intervention in the plot. In this paper, the authors describe a Mixed Reality system based on a “magic mirror” model (Figure 1), in which the user’s image is captured in real time by a video camera, extracted from his/her background and mixed with a 3D graphic model of a virtual stage including the synthetic characters taking part in the story. The resulting image is projected on a large screen facing the user, who sees his/her own image embedded in the virtual stage with the synthetic actors. The graphic component of the Mixed Reality world is based on a game engine, Unreal Tournament 2003TM. This engine not only performs graphic rendering and character animation but incorporates a new version of our previously described storytelling engine [1]. A single 2D camera facing the user analyses the image in real-time by segmenting the user’s contours [2]. The objective behind segmentation is twofold. Firstly, it extracts the image silhouette of the user in order to be able to inject it into the virtual setting on the projection screen. Secondly, it can recognise user behaviour, including symbolic gestures, in real-time, hence supporting a new interaction channel.