A Motion-Based Capture and Replay Platform of Mobile Device Interaction

Learn more
example remotion hardware

Remotely mirror the phone interactions of your users

Remotion enables richer context sharing in remote users, especially during usability studies. Better understand users' grip, attention, emotion, posture, and habits. This website provides instructions for setting up and deploying Remotion, and hosts the open source software and blueprints.

What is Remotion for?

Remotion is a new way of replaying behavior of people visiting your website or application.

A remote user's hand motions reflect contextual information about their surroundings, point of attention, and even signals particular emotions and intent. Imagine a scenario when a user shopping online on a mobile device proceeds to the checkout page which asks for their credit card information. Some users will put down the phone and start to look for their wallet, while others may have the cards readily available without the need to place the phone away. None of this contextual information is captured on the screen, yet the different movements resulting from how a user handles their device reveal more about the situation than what is displayed on the screen.

Remotion extracts more information about a user's performance in a remote usability testing, without the privacy or data size issues of video recordings. Remote user studies can feel as if the user is invisible but in the room manipulating the device, through a software or hardware replay. The mobile device is shown as a 3D model (software) or controlled by a custom robotic mount (hardware) to simulate exactly how the device was manipulated and what was shown on the screen.


How Remotion Works

Remotion is comprised of five components:

The client software library for motion sensing on mobile devices (e.g., Android phones and Surface tablets) enables developers to collect sensing data by simply deploying an application on the participants' phone.

The screen projection component captures what the user sees on their screen and projects it to the screen in a software or hardware replay.

The lightweight server receives and organizes the data collected with Remotion's client and Remotion's screen provides an interface to control the software and hardware visualization replay.

The software visualization displays on the experimenter's screen a replay of what the user sees on their screen and the phone's movements.

Remotion's hardware (physical) replay replicates the remote user's movement and screen on a one-to-one scale. The robotic mount (with 4-axis degrees of freedom) is offered as blueprints for at-home construction.

Remotion system architecture diagram

Paper and Demo

Remotion: A Motion-Based Capture and Replay Platform of Mobile Device Interaction for Remote Usability Testing
Jing Qian, Arielle Chapin, Alexandra Papoutsaki, Fumeng Yang, Klaas Nelissen, Jeff Huang. Remotion: A Motion-Based Capture and Replay Platform of Mobile Device Interaction for Remote Usability Testing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 2(2), Article 77.

Blueprints, Source Code and Documentation

We'd love for you to also have your own Remotion setup. Hardware models, video instructions, and source code are all open source. The files necessary for setting up Remotion are on Github.

Remotion Github repository

The installation video is also available on Youtube.

Research Team

Arielle Chapin

Alexandra Papoutsaki

Fumeng Yang

Klaas Nelissen

Funded in part by the National Science Foundation IIS-1552663