Home im.ve Group im.ve Lab Projects Publications Teaching News, Events, Activities Contact us News

Functional Reactive Virtual Reality


FRVR Design

FRVR is designed to take advantage of the expressive power of the FRP paradigm for the programming of the essence of Dynamic, Interactive VEs. FRP focuses on the creation and specification continuous behaviors and the interactions of those behaviors. The FRP system is specialized to handle this class of problems. In the same way, VR systems are specialized to handle the technical issues of driving immersive VR.  VR systems are classically good at abstraction from the complication of the hardware systems that are required for such systems and provide support for graphics display. The combination of  FRP with a VR system enables the creation of persuasive dynamic, interactive VEs in VR.

FRVR is built around providing a loose coupling of the FRP system and a VR system. This provides a number of benefits, including that a framework of support is build for the integration of FRP with most any VR system.  The basic architecture of the FRVR design  can be seen in the diagram below.  In this diagram the Yampa FRP system is used and VR Juggler provides the VR framework capabilities. This allows connection with any SG supported in VR Juggler. Because of the loose coupling, integration into other VR systems is easy as demonstrated by the existing connection to the AVANGO system.

 


Part of the challenge of incorporating the two systems is that the most up to date FRP system, Yampa, is programmed in the Haskell language. This creates unique challenges as the C++ based VR system and Haskell based Yampa have to communicate. In order to bridge the two systems, the current implementation of FRVR uses a custom tagged, shared memory system.  This tagged memory system is reminiscent of the shared memory (not distributed agents) BlackBoard systems used in some robotics projects. The shared memory system also helps maintain the abstraction between the VR system and Yampa systems. Built into the shared memory system is another component of the challenge of a multi-language system, the marshalling of data types. This is required, as the two languages use different primitive data types.  To simplify the exchange only the standard computer graphic mathematics types are presently supported.  The types supported are those of the GMTL (Graphics Math Template Language) project, one of the only independent math libraries and the one VR Juggler uses. A CG math library that mostly matchs GMTL was developed  for Haskell as part of the FRVR module.

The DIVE author performs work on both the FRP and VR sides. The graphical system is still controlled by the VR side, typically using a Scene Graph system. Crucial for the functioning of FRVR is that the author connects the dynamics information produced in the FRP system to the appropriate place in the scene graph.  This typically involves simply moving position and/or rotation information from the simulation to the appropriate transformation node in the SG.  For interaction, the author must also assure that the VR system delivers required information to the FRP side, e.g. wand positions, events from "button presses", and selection and collision information. The insertion of input data into the shared memory occurs just before a call to the simulation and the outputs of the simulation are routed to the SG directly after that call. The simulation is triggered by a single FRVR provided function. When using a more advance VR system, such as AVANGO, this routing work can be simplified by usage of mechanisms like the Field Connections of AVANGO. After setup, data is automatically kept up to date. Below is a diagram of how this works in the AVANGO FRVR binding.

 

Here we see how the FRVR system works with the AVANGO VR system.  AVANGO's data-flow system is used to deliver values to the appropriate place in the scene graph.  Wand position and orientation information is delivered to the FRP system as well as button press events to enable interaction.


The author does the majority of their work in creating dynamic interactive environments on the FRP side. Their work is assisted by the FRVR Haskell side implementation.  A number of tools and extensions are included in the FRVR module to ease the development. The connection to the shared memory and dealing with the conversion of data types is handled by special module portions.  Functions to simplify the setup of the simulation environment are also included.  A math library for computer graphics math types makes dealing with standard CG methods possible. Even support for quaternion integration is included. Finally a number of extensions to the FRP libraries are required to make FRP work in the VR world. A specialized system kernel for usage with VR is provided.  Often used functionalities and extensions to the Yampa toolset that simplify programming of standard concepts are provided. Finally, a library of examples shows how FRP can be used in VR.

More thorough descriptions of the design can be found in the publications below:

Functional Reactive Virtual Reality

Blom K., Beckhaus S.
in Short Paper Proceedings of IPT-EGVE Symposium (2007), B. Fröhlich, R. Blach, and R. van Liere (Editors), Weimar, pg. 295-302
File(s): Paper (PDF)


Integrating Functional Reactive Programming in a High-Level VR Framework

Blom K., Beckhaus S.
Virtuelle und Erweiterte Realität, 4. Workshop der GI-Fachgruppe VR/AR", M. E. Latoschik, B. Fröhlich (Editors), Weimar, pg. 189-196, ISBN 978-3-8322-6367-6
File(s): Paper (PDF)



⇒ back to FRVR webpage