Ben Smith’s Musiverse

Research at eDREAM and IACAT (Creativity and Computing theme) has several long-term goals in keeping with their overall institutional missions to spur innovation in art and science.  Their primary current research is in the following areas:

• Realtime Interactive High Performance Computing
• High Bandwidth Low-Latency Networking for the Performing and Collaborative Arts
• Scientific Visualization and Sonification
• Multiuser, Distributed Virtual Worlds
• Ubiquitous Interactive Media

Realtime, Interactive HPC

Most High Performance Computing (HPC) operates in a batch processing mode. For artists, particularly performing artists, there is a need to interact directly and in a timely fashion with the processing to allow sufficient room for intuitive and aesthetic responses to be brought to bear.

The realtime interactive goals can be outlined as follows:

• take high-bandwidth inputs from a user or multiple networked users—in the form of multimodal gestural inputs
• pre-process those inputs using signal processing and machine learning
• feed the results of the pre-processing into the simulation or other computation to affect its course
• get results out that can be visualized and sonified—or represented in other ways such as actuators, robotics, etc.
• all within a few milliseconds so that it appears to the human user that there is no delay in the system response.

The high-performance goals are in keeping with existing uses of HPC:

• scalable computations through virtualization
• large scale parallel computation (hundreds or thousands of nodes)
• access to large data sets (tera- to petabytes)
• high bandwidth connectivity for inputs and outputs such as to a display (10 gb pipeline)

Realtime interactions with large scale computations will impact many areas of computation and human interaction with computations. These include at least:

• interactive simulations and steerable computations
• multimodal HCI (gesture control, video, etc.)
• machine learning inputs from human users
• exploratory research modes
• simultaneous collaborative, multiuser simulation control

Realtime, Interactive HPC Projects and Applications

The current projects that are helping to drive research in realtime interactive HPC include:

• mWorlds virtual worlds simulations (Garnett)
• BioSim, an mWorlds implementation of disease transmission (Garnett, Yahja)
• interactive environmental change simulation (Garnett, Harris, Wuebbles)

High-Bandwidth, Low-Latency Networking

Performing arts ensembles, whether dance or music or theater, rely frequently on precise, repeatable, split-second reactions and responses between multiple performers. Typical computer networks, on the other hand, have latencies on the order of 100s of milliseconds to seconds while their inherent jitter makes prediction all but humanly impossible. New networks are coming soon that will enable performers to interact over continental scale distances with the required low latency and with uniform or very minimal jitter. We are developing systems now to lead the way in such networks to enable distant performers to collaborate both in rehearsal and in performance.

Networking research goals include:

• uncompressed bi-directional HD capable (1.8 gb/s times 2)
• with < 100 ms latency
• between any two points across continents

High-Bandwidth, Low-Latency Networking Projects and Applications

• mWorlds virtual worlds simulations (Garnett)
• musiverse performances (Garnett, Smith)
• Telematic Performances (Garnett, UIPUI, RPI)
• Digital Collective (Garnett and students)
• Astral Connected (Toenjes, et al)

AVL’s tornado rendered for digital fulldome display

Scientific Visualization and Sonification

The visual and sonic arts provide a means to enhance and interact with science data and scientific processes that can significantly increase their human impact on the scientists themselves, aiding in the knowledge discovery process, and on the wider public to influence policy and increase education.

Scientific Visualization and Sonification Projects and Applications

• vMaya (AVL)
• mWorlds virtual worlds simulations (Garnett)
• Interactive environmental change simulation (Garnett, Harris, Wuebbles, McGrath, Craig)
• BioSim, an mWorlds implementation of disease transmission (Garnett, Yahja)

Distributed, Multi-user Virtual Worlds

Virtual worlds are still in their infancy. They are difficult to create, impossible for non-professionals, and even more difficult to scale to large numbers of users and objects. Our work is creating a framework for virtual world development, mWorlds, that will ultimately be easy for end users to use to develop and support their own virtual worlds. It is highly distributed and highly modular.

Distributed Multi-user Virtual Worlds Projects and Applications

• mWorlds virtual worlds simulations (Garnett)
• BioSim, an mWorlds implementation of disease transmission (Garnett, Yahja)
• interactive environmental change simulation (Garnett, Harris, Wuebbles)
• Musiverse performances (Garnett, Smith)
• Telematic Performances (Garnett, UIPUI, RPI)
• Digital Collective (Garnett and students)

Ubiquitous, Interactive Media

Human-media interactions are becoming increasingly unglued from the particularities of place. They are increasingly wireless, distributed, etc. The real world is also becoming increasingly suffused with the virtual world, the real is augmented by the virtual, the virtual is extended further into the ubiquitous real. Ubiquitous Interactive Media transcend the boundaries of place, of representation, of mediation and enable new modes of interactive engagement with information and digital processes.

Ubiquitous Interactive Media Projects and Applications

• Astral Connected (Toenjes, et al)
• Musiverse performances (Garnett, Smith)
• Telematic Performances (Garnett, UIPUI, RPI)
• Digital Collective (Garnett and students)